Oct 30 21:39:47.108317 ip-10-0-128-19 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:39:47.108330 ip-10-0-128-19 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:39:47.108339 ip-10-0-128-19 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:39:47.108659 ip-10-0-128-19 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:39:58.482347 ip-10-0-128-19 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:39:58.482365 ip-10-0-128-19 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2495ca1f50044aa98b21507af91242bc --
Oct 30 21:41:46.677056 ip-10-0-128-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:41:47.083533 ip-10-0-128-19 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.083533 ip-10-0-128-19 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:41:47.083533 ip-10-0-128-19 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.083533 ip-10-0-128-19 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.087247 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087170    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.087247 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087246    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.087247 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087250    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087253    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087256    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087259    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087261    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087265    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087267    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087270    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087272    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087275    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087278    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087280    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087282    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087285    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087288    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087291    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087293    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087296    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087299    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.087341 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087302    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087304    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087307    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087310    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087312    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087315    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087317    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087320    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087322    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087325    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087330    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087333    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087336    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087339    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087342    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087345    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087348    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087352    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087355    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.087791 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087358    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087361    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087364    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087368    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087372    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087375    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087379    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087382    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087385    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087387    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087389    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087392    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087395    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087397    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087400    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087402    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087405    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087408    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087410    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.088241 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087412    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087415    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087417    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087420    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087422    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087425    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087427    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087430    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087433    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087435    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087438    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087441    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087444    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087447    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087450    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087452    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087454    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087457    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087459    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087462    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.088679 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087464    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087467    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087469    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087472    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087474    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087477    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087479    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087482    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087485    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087487    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087490    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087492    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087494    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087497    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087499    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087502    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087504    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087507    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087509    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.089192 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087512    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087515    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087517    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087519    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087522    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087526    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087528    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.087532    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089611    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089620    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089623    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089626    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089629    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089632    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089635    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089637    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089640    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089643    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089645    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089648    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.089656 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089650    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089653    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089656    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089658    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089661    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089665    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089669    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089673    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089675    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089678    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089681    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089683    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089686    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089689    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089691    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089693    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089696    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089698    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089701    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.090184 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089703    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089706    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089709    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089712    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089714    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089716    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089719    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089721    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089724    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089726    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089729    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089731    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089733    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089736    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089738    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089743    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089746    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089748    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089751    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089753    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.090637 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089756    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089759    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089761    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089764    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089766    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089769    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089785    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089788    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089790    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089793    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089795    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089798    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089801    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089803    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089806    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089809    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089812    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089815    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089817    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.091132 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089820    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089823    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089825    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089828    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089830    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089833    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089835    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089838    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089841    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089847    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089851    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089854    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089857    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089860    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089862    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089865    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089867    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089870    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089872    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089875    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.091578 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089878    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089880    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089883    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089886    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089888    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089891    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089893    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089896    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089898    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089901    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089903    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089906    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089909    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089911    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089914    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.089916    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090429    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090434    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090437    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090440    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.092097 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090443    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090446    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090448    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090451    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090454    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090456    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090459    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090462    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090464    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090467    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090470    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090473    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090476    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090478    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090481    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090483    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090486    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090488    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090491    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090493    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.092565 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090496    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090499    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090502    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090504    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090507    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090509    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090512    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090514    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090517    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090519    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090522    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090525    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090527    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090530    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090533    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090535    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090537    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090540    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090543    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090545    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.093063 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090548    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090551    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090553    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090556    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090561    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090564    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090566    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090569    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090571    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090574    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090576    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090579    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090581    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090584    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090586    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090589    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090592    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090594    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090597    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.093533 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090599    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090602    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090604    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090607    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090610    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090612    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090615    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090618    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090621    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090623    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090625    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090628    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090630    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090633    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090635    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090638    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090640    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090643    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090646    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.094008 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090649    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090651    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090654    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090657    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090659    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090662    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090664    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090667    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090669    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090672    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090674    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090677    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090680    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090682    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090685    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090688    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090691    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090693    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090696    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.094457 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090700    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090703    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090707    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090711    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.090714    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091820    2584 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091830    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091838    2584 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091843    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091848    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091851    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091858    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091862    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091866    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091869    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091872    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091876    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091879    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091881    2584 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091884    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091888    2584 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091890    2584 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091894    2584 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:41:47.094948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091898    2584 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091901    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091904    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091908    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091911    2584 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091915    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091918    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091922    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091926    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091929    2584 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091932    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091935    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091938    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091941    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091945    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091948    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091951    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091954    2584 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091957    2584 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091960    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091965    2584 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091969    2584 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091971    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091974    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:41:47.095467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091977    2584 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091982    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091985    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091988    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091991    2584 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091994    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.091997    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092000    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092003    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092006    2584 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092009    2584 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092012    2584 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092018    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092021    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092025    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092028    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092031    2584 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092034    2584 flags.go:64] FLAG: --help="false"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092037    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092040    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092043    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092046    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092050    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:41:47.096086 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092054    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092056    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092059    2584 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092062    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092065    2584 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092068    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092071    2584 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092074    2584 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092077    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092080    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092083    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092085    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092088    2584 flags.go:64] FLAG: --lock-file=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092091    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092094    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092097    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092102    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092105    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092109    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092112    2584 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092115    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092120    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092123    2584 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092126    2584 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092130    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:41:47.096630 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092133    2584 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092136    2584 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092139    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092142    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092145    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092148    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092151    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092154    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092157    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092164    2584 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092167    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092170    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092173    2584 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092176    2584 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092179    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092182    2584 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092185    2584 flags.go:64] FLAG: --port="10250"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092188    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092191    2584 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-070da6bf92d26b853"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092194    2584 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092197    2584 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092200    2584 flags.go:64] FLAG: --register-node="true"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092202    2584 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092206    2584 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092209    2584 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092212    2584 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:41:47.097295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092215    2584 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092220    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092225    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092228    2584 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092231    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092233    2584 flags.go:64] FLAG: --runonce="false"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092236    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092239    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092242    2584 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092245    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092248    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092251    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092254    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092258    2584 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092261    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092264    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092267    2584 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092270    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092273    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092275    2584 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092278    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092283    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092286    2584 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092289    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092293    2584 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:41:47.097923 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092296    2584 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092299    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092301    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092304    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092307    2584 flags.go:64] FLAG: --v="2"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092311    2584 flags.go:64] FLAG: --version="false"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092316    2584 flags.go:64] FLAG: --vmodule=""
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092320    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092325    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092450    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092454    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092457    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092460    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092463    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092465    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092468    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092471    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092474    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092477    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092480    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092482    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.098554 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092485    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092487    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092490    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092493    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092495    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092497    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092500    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092503    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092505    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092508    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092511    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092513    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092516    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092518    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092521    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092523    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092526    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092529    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092531    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.099065 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092535    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092539    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092542    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092545    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092547    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092550    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092552    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092555    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092558    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092560    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092562    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092565    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092568    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092570    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092573    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092576    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092578    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092581    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092583    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.099513 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092586    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092588    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092591    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092594    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092596    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092599    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092601    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092604    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092606    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092609    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092612    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092614    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092616    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092622    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092625    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092628    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092631    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092634    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092636    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092639    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.099971 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092641    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092644    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092646    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092649    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092652    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092654    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092657    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092659    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092662    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092664    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092667    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092670    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092672    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092676    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092679    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092682    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092685    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092688    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092691    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092694    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.100440 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092696    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092699    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092702    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092704    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092707    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092711    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092714    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092718    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092726    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092729    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092732    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092734    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092737    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092740    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092742    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092745    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.101000 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.092786    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092829    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092833    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092837    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092839    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092843    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092845    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092848    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092851    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092854    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092856    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092859    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092862    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092864    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092867    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092869    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092872    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092875    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092877    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092879    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.101377 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092882    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092885    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092891    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092895    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092898    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092902    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092905    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092908    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092910    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092913    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092916    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092919    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092921    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092924    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092927    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092929    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092933    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092936    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092939    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.101853 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092942    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092944    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092947    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092949    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092952    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092955    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092957    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092960    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092963    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092966    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092968    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092971    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092973    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092976    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092978    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092982    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092985    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092987    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092990    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092993    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.102297 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092996    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.092998    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093001    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093003    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093006    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093008    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093018    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093022    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093025    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093028    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093030    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093033    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093036    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093038    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093041    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093043    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093046    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093048    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093051    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.102757 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093054    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093056    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093059    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093061    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093064    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093067    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093070    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093072    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093076    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093078    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093081    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093084    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093086    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093089    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093092    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093094    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093097    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093100    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093102    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.103206 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093105    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093108    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093111    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093115    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093118    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093120    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093123    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093125    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093128    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.093130    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.099566    2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.099581    2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.099597    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.099601    2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099663    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099668    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099671    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099674    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099678    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099681    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.103682 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099684    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099687    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099690    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099693    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099695    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099698    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099701    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099704    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099707    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099710    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099712    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099715    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099718    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099720    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099723    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099725    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099728    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099730    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099733    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099735    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.104213 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099738    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099740    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099743    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099747    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099750    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099753    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099755    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099758    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099761    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099763    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099766    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099768    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099786    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099790    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099792    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099795    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099798    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099800    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099803    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099806    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.104670 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099809    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099813    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099818    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099822    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099825    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099829    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099832    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099835    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099838    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099841    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099843    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099846    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099848    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099851    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099853    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099856    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099859    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099861    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099864    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.105167 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099867    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099869    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099871    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099874    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099877    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099879    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099882    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099884    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099887    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099889    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099892    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099894    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099897    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099900    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099903    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099905    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099908    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099910    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099912    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099915    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.105600 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099918    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099920    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099923    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099925    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099928    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099931    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099933    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099936    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099938    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099941    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099943    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099946    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099948    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099951    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099953    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099956    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099958    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099961    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099963    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.106082 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099965    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.099968    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.099994    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100124    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100128    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100131    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100133    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100137    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100139    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100142    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100145    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100147    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100150    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100153    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100155    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100158    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.106512 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100160    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100163    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100165    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100168    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100170    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100173    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100175    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100178    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100180    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100183    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100185    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100188    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100190    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100193    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100195    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100198    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100200    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100203    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100205    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100208    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.106919 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100210    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100213    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100216    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100218    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100221    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100224    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100228    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100232    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100236    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100239    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100242    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100245    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100248    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100250    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100253    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100255    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100258    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100260    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100263    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.107379 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100266    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100268    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100271    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100273    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100276    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100278    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100281    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100283    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100286    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100288    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100291    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100293    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100296    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100298    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100304    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100307    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100309    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100312    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100315    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.107884 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100318    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100321    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100323    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100326    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100328    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100331    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100333    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100336    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100339    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100341    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100344    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100346    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100349    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100351    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100354    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100356    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100359    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100361    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100364    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100366    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.108375 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100369    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100371    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100374    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100376    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100379    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100381    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100384    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100388    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100390    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100393    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100395    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100398    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100401    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100404    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100406    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.109157 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.100428    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100460    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100465    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100467    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100470    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100472    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100475    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100477    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100480    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100483    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100485    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100488    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100490    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100493    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100496    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100498    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100500    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100503    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100506    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100508    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.109520 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100511    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100513    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100516    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100519    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100523    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100526    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100528    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100531    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100533    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100538    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100541    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100544    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100546    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100549    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100551    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100554    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100557    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100559    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100562    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100564    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.110029 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100567    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100569    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100572    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100575    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100577    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100580    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100582    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100584    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100587    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100589    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100592    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100594    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100597    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100599    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100602    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100604    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100608    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100611    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100613    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.110524 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100616    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100618    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100622    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100626    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100629    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100632    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100634    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100637    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100640    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100642    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100645    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100648    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100650    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100653    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100655    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100658    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100660    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100663    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100665    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.111160 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100668    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100671    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100673    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100676    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100678    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100681    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100683    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100686    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100688    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100691    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100694    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100697    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100700    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100702    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100705    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100709    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100711    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100714    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100717    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100721    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.111592 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100724    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100728    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100731    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100735    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100738    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100741    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100743    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100746    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:47.100748    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.100814    2584 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.102817    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.103971    2584 server.go:1014] "Starting client certificate rotation"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.104068    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.104103    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.107097    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:47.112102 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.111735    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:47.120351 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.120333    2584 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:41:47.126505 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.126491    2584 log.go:25] "Validated CRI v1 image API"
Oct 30 21:41:47.127331 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.127307    2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:41:47.131943 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.131921    2584 fs.go:135] Filesystem UUIDs: map[6884a548-4cde-47cd-ad52-8041eacac26a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ab474186-8d83-4259-93af-14b0f0d62ae4:/dev/nvme0n1p3]
Oct 30 21:41:47.132026 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.131941    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:41:47.136966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.136851    2584 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:47.135574118 +0000 UTC m=+0.342965582 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100483 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec267ac6070a302663a68f3eda6ed304 SystemUUID:ec267ac6-070a-3026-63a6-8f3eda6ed304 BootID:2495ca1f-5004-4aa9-8b21-507af91242bc Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:75:0f:e9:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:75:0f:e9:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:57:11:bc:42:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:41:47.136966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.136963    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:41:47.137129 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.137068    2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:41:47.139813 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139789    2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:41:47.139933 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139814    2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-19.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:41:47.139974 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139935    2584 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:41:47.139974 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139943    2584 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:41:47.140033 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139977    2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:47.140033 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.139993    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:47.141028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.141018    2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:41:47.144762 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.144751    2584 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:41:47.144879 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.144870    2584 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:41:47.145706 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.145697    2584 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:41:47.145746 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.145710    2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:41:47.145746 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.145723    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:41:47.145746 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.145732    2584 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:41:47.145746 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.145740    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:41:47.146045 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.146035    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:47.148180 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.148155    2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:41:47.156567 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.156550    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:47.157399 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.157387    2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:41:47.157438 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.157416    2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:41:47.158519 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158508    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158525    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158532    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158538    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158544    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158550    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158558    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:41:47.158561 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158563    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:41:47.158736 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158571    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:41:47.158736 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158576    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:41:47.158736 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158584    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:41:47.158736 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.158593    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:41:47.159457 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.159448    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:41:47.159486 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.159458    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:41:47.163164 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.163153    2584 server.go:1263] "Started kubelet"
Oct 30 21:41:47.163441 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.163218    2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:41:47.163441 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.163253    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:41:47.163441 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.163346    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:41:47.163878 ip-10-0-128-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:41:47.164246 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.164232    2584 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:41:47.166645 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.166567    2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:41:47.168646 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.168628    2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:41:47.172624 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.172605    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:47.173935 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.173920    2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:41:47.174061 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.174036    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found"
Oct 30 21:41:47.174661 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174645    2584 factory.go:55] Registering systemd factory
Oct 30 21:41:47.174661 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174662    2584 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:41:47.174821 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174735    2584 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:41:47.174821 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174752    2584 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:41:47.174821 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174735    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:41:47.174949 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174856    2584 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:41:47.174949 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174864    2584 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:41:47.174949 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174868    2584 factory.go:153] Registering CRI-O factory
Oct 30 21:41:47.174949 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174883    2584 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:41:47.174949 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174935    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:41:47.175126 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174952    2584 factory.go:103] Registering Raw factory
Oct 30 21:41:47.175126 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.174965    2584 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:41:47.176069 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.176054    2584 manager.go:319] Starting recovery of all containers
Oct 30 21:41:47.181822 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.181798    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:41:47.188157 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.188142    2584 manager.go:324] Recovery completed
Oct 30 21:41:47.193164 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.193150    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.194617 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.194593    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.194684 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.194632    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.194684 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.194646    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.195163 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195151    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.195163 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195156    2584 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:41:47.195294 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195165    2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:41:47.195294 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195184    2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:41:47.195817 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195802    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.195875 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195829    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.195875 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.195842    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.197332 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.197318    2584 policy_none.go:50] "Start"
Oct 30 21:41:47.197393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.197335    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:41:47.197393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.197349    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:41:47.199295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.199276    2584 policy_none.go:44] "Start"
Oct 30 21:41:47.225401 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.225365    2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:41:47.228074 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.227921    2584 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:41:47.228148 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.228095    2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:41:47.228148 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228106    2584 server.go:87] "Starting device plugin registration server"
Oct 30 21:41:47.228384 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228373    2584 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:41:47.228418 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228388    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:41:47.228593 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228479    2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:41:47.228593 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228563    2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:41:47.228593 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.228571    2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:41:47.229135 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.229113    2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:41:47.229197 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.229168    2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-19.us-west-1.compute.internal\" not found"
Oct 30 21:41:47.303669 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.303649    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-19.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:41:47.303790 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.303642    2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-19.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:41:47.304862 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.303650    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-19.us-west-1.compute.internal.18e36c9f16e26d51  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-19.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-19.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47.163127121 +0000 UTC m=+0.370518586,LastTimestamp:2026-10-30 21:41:47.163127121 +0000 UTC m=+0.370518586,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-19.us-west-1.compute.internal,}"
Oct 30 21:41:47.328642 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.328623    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.329440 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.329422    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.329553 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.329453    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.329553 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.329473    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.329553 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.329515    2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.382942 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.382840    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:47.382942 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.382876    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8cjqp"
Oct 30 21:41:47.425646 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.425624    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:41:47.425646 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.425646    2584 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:41:47.427588 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.425664    2584 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:41:47.427588 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.425692    2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:41:47.460006 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.459983    2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.525899 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.525878    2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"]
Oct 30 21:41:47.525976 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.525942    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.526745 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.526727    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.526837 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.526754    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.526837 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.526766    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.528910 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.528899    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.529062 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.529101 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529080    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.529595 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529576    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.529676 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529608    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.529676 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529618    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.529676 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529576    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.529799 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529682    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.529799 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.529694    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.531763 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.531737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.531856 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.531791    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.532536 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.532518    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.532628 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.532549    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.532628 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.532559    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.559408 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.559393    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.560997 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.560970    2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.561471 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.561448    2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.561963 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.561949    2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.562008 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.561970    2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-19.us-west-1.compute.internal\": node \"ip-10-0-128-19.us-west-1.compute.internal\" not found"
Oct 30 21:41:47.566406 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.566389    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.568862 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:47.568848    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.576498 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.576482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.576579 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.576518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.576579 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.576545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.674650 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.674598    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676687 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676792 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676792 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676792 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676758    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676792 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676763    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.676920 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.676762    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.735421 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.735404    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:47.735496 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.735486    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.790263 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.790248    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:47.869328 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.869313    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:47.871455 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:47.871439    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:41:48.103631 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.103567    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:41:48.104152 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.103674    2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:41:48.104152 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.103699    2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:41:48.104152 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.103684    2584 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3ee7570faedf4a5ca3084ec2df71db2-8c14c679226562c9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-19.us-west-1.compute.internal.18e36c9f18c36c9f\": read tcp 10.0.128.19:57256->18.219.234.48:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-19.us-west-1.compute.internal.18e36c9f18c36c9f  default   5033 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-19.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-19.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-19.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47 +0000 UTC,LastTimestamp:2026-10-30 21:41:47.329479487 +0000 UTC m=+0.536870955,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-19.us-west-1.compute.internal,}"
Oct 30 21:41:48.146427 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.146407    2584 apiserver.go:52] "Watching apiserver"
Oct 30 21:41:48.173417 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.173389    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:48.387308 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.387237    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:47 +0000 UTC" deadline="2028-05-20 00:48:48.374738036 +0000 UTC"
Oct 30 21:41:48.387308 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.387262    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13611h6m59.987478538s"
Oct 30 21:41:48.388886 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.388872    2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:41:48.389128 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.389114    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh","openshift-multus/multus-additional-cni-plugins-vzz5n","openshift-multus/multus-fttwq","openshift-network-diagnostics/network-check-target-pjnlr","kube-system/konnectivity-agent-9l2mj","openshift-dns/node-resolver-st48b","openshift-network-operator/iptables-alerter-n52cs","kube-system/global-pull-secret-syncer-b7t59","openshift-image-registry/node-ca-q2gnf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-qltrb","kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-fxqp4","openshift-multus/network-metrics-daemon-q2bzm"]
Oct 30 21:41:48.391285 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.391272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.393184 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.393164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.395146 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.395131    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.397061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.397050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:48.397129 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.397113    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:48.399023 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.399005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.400966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.400947    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.402966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.402952    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.407163 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.405228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.407163 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.405305    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:48.407567 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.407550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.410474 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.410459    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.412541 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.412527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.414522 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.414508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.414590 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.414552    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:48.441818 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.441805    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.441961 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.441944    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.442454 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.442442    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:41:48.442531 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.442522    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:41:48.442684 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.442673    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:41:48.442810 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.442796    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.442846 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.442834    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.446491 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.446479    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:41:48.446529 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.446485    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\""
Oct 30 21:41:48.449765 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.449752    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:41:48.476256 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.476240    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:41:48.480795 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlppm\" (UniqueName: \"kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.480857 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.480857 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.480857 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480836    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.480964 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.480964 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.480964 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.481077 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481077 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.480988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481077 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481077 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481242 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txs9p\" (UniqueName: \"kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.481414 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwxd\" (UniqueName: \"kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481475    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481489    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.481668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481663    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45x8s\" (UniqueName: \"kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481791    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.481985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482183    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhr7\" (UniqueName: \"kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvn84\" (UniqueName: \"kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7sz\" (UniqueName: \"kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.482571 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prd5v\" (UniqueName: \"kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482655    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfls\" (UniqueName: \"kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482764    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.483028 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.482848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.538467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538444    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\""
Oct 30 21:41:48.538467 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538453    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\""
Oct 30 21:41:48.538639 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538626    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\""
Oct 30 21:41:48.538711 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538696    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.538953 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538940    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:41:48.539014 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538971    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:41:48.539014 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.538991    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\""
Oct 30 21:41:48.539106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539021    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:48.539106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539036    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:41:48.539106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539039    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:41:48.539106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539069    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\""
Oct 30 21:41:48.539106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539098    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gzgdd"
Oct 30 21:41:48.539928 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539907    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\""
Oct 30 21:41:48.540005 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.539967    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\""
Oct 30 21:41:48.540052 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540040    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.540274 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540258    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\""
Oct 30 21:41:48.540328 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540299    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.540421 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540407    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.540466 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540439    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.540555 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540542    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.540641 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540561    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:41:48.540641 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540571    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:41:48.540641 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540582    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.540786 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540645    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.540786 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540757    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.540866 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540806    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.540866 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.540813    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:41:48.547140 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.547126    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:41:48.583393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583365    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.583393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583397    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlppm\" (UniqueName: \"kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.583513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583463    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.583536    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583592    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.583603    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:49.083584269 +0000 UTC m=+2.290975744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583605    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583606    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583628    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583675    2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.583793 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583810    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583823    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583852    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583914    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583952    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.583993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584005    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584004    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584047    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584049    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txs9p\" (UniqueName: \"kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584050    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.584398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584196    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584204    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584255    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwxd\" (UniqueName: \"kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584342    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584352    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584295    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.585215 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584398    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584483    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584544    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584587    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584593    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584663    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584700    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584545    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45x8s\" (UniqueName: \"kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584728    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.586029 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584807    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584829    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584851    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584904    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584930    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.586811 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585154    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585167    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585206    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585233    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585255    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585285    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhr7\" (UniqueName: \"kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585307    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585321    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvn84\" (UniqueName: \"kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.587550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7sz\" (UniqueName: \"kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd5v\" (UniqueName: \"kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585634    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585642    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585353    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585685    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585700    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588138 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585732    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585750    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585800    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585812    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.584931    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.585894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfls\" (UniqueName: \"kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.585905    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.585960    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:49.085945622 +0000 UTC m=+2.293337095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586011    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586109    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586132    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586149    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586188    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586194    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586236    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.588627 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586278    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586287    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586341    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586378    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586437    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586440    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.586490    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.587233    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.587383    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.587499    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.587514    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.589145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.587631    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.637031 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.637005    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txs9p\" (UniqueName: \"kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.637106 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.637008    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlppm\" (UniqueName: \"kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.640174 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.640111    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:48.640174 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.640133    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:48.640174 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.640145    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:48.640406 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:48.640204    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:41:49.140189177 +0000 UTC m=+2.347580650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:48.640881 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.640859    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45x8s\" (UniqueName: \"kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.641504 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.641480    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwxd\" (UniqueName: \"kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.641659 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.641633    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvn84\" (UniqueName: \"kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.642219 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.642194    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7sz\" (UniqueName: \"kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.642308 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.642216    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhr7\" (UniqueName: \"kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.642640 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.642622    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd5v\" (UniqueName: \"kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.643200 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.643183    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfls\" (UniqueName: \"kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:48.717858 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.717839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:41:48.736139 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.736121    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:41:48.739849 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.739834    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fttwq"
Oct 30 21:41:48.754221 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.754201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:48.763830 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.763814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-st48b"
Oct 30 21:41:48.783289 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.783269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:41:48.798436 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.798411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:41:48.819181 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.819157    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:41:48.826881 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.826863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:41:48.875422 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:48.875385    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf36e468c742c0445d905ae4f02392b0.slice/crio-392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891 WatchSource:0}: Error finding container 392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891: Status 404 returned error can't find the container with id 392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891
Oct 30 21:41:48.875869 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:48.875851    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4301f18ed471dd47244e9704246a79.slice/crio-2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9 WatchSource:0}: Error finding container 2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9: Status 404 returned error can't find the container with id 2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9
Oct 30 21:41:48.880076 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:48.880063    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:41:49.089390 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.089315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:49.089390 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.089360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:49.089553 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.089437    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:49.089553 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.089449    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:49.089553 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.089480    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.089469549 +0000 UTC m=+3.296861004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:49.089553 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.089496    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.089485483 +0000 UTC m=+3.296876938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:49.189874 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.189841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:49.190429 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.189976    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:49.190429 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.189996    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:49.190429 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.190006    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:49.190429 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:49.190046    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.190033938 +0000 UTC m=+3.397425403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:49.252025 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.252000    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.429299 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.429216    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerStarted","Data":"392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891"}
Oct 30 21:41:49.430197 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.430177    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"fe4301f18ed471dd47244e9704246a79","Type":"ContainerStarted","Data":"2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9"}
Oct 30 21:41:49.484929 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.484898    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7524bf5c_af7d_4d94_a660_06fe000da0ae.slice/crio-c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b WatchSource:0}: Error finding container c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b: Status 404 returned error can't find the container with id c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b
Oct 30 21:41:49.494052 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.494031    2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.509672 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.509649    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7040152_696f_44aa_b210_81c5093eed9d.slice/crio-7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046 WatchSource:0}: Error finding container 7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046: Status 404 returned error can't find the container with id 7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046
Oct 30 21:41:49.542850 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.542814    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:48 +0000 UTC" deadline="2028-03-30 22:02:24.383859842 +0000 UTC"
Oct 30 21:41:49.542850 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:49.542848    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12408h20m34.841014635s"
Oct 30 21:41:49.557218 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.557197    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7dc4c7_b8fb_44e8_8934_c44ef5a33634.slice/crio-5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09 WatchSource:0}: Error finding container 5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09: Status 404 returned error can't find the container with id 5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09
Oct 30 21:41:49.561137 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.561115    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96a8d72_5f70_4685_9ade_378699d4d95e.slice/crio-a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62 WatchSource:0}: Error finding container a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62: Status 404 returned error can't find the container with id a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62
Oct 30 21:41:49.561673 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.561656    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de5e171_5eb9_4628_806b_11b5fc875413.slice/crio-473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0 WatchSource:0}: Error finding container 473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0: Status 404 returned error can't find the container with id 473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0
Oct 30 21:41:49.609978 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.609944    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4300224c_3e21_45f0_bf6b_d60a935f3aea.slice/crio-398aa4afd2430154323ff86a995f1a8b032754505913c6129af72fcdcdc0c0b9 WatchSource:0}: Error finding container 398aa4afd2430154323ff86a995f1a8b032754505913c6129af72fcdcdc0c0b9: Status 404 returned error can't find the container with id 398aa4afd2430154323ff86a995f1a8b032754505913c6129af72fcdcdc0c0b9
Oct 30 21:41:49.647674 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.647642    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9222687b_1080_41af_8c30_776403f2c1ec.slice/crio-bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe WatchSource:0}: Error finding container bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe: Status 404 returned error can't find the container with id bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe
Oct 30 21:41:49.699416 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.699393    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ada66f_5869_428a_973b_91d299646056.slice/crio-a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb WatchSource:0}: Error finding container a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb: Status 404 returned error can't find the container with id a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb
Oct 30 21:41:49.719173 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:41:49.719154    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee19afb5_85b7_4f2a_b0ae_1a848e32d728.slice/crio-416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d WatchSource:0}: Error finding container 416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d: Status 404 returned error can't find the container with id 416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d
Oct 30 21:41:50.097004 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.096923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:50.097166 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.097063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:50.097166 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.097073    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.097166 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.097140    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.097120915 +0000 UTC m=+5.304512387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.097339 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.097219    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.097339 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.097261    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.097248223 +0000 UTC m=+5.304639687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.197679 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.197638    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:50.198335 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.197848    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:50.198335 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.197869    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:50.198335 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.197881    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.198335 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.197942    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.197923719 +0000 UTC m=+5.405315172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.426320 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.426235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:50.426543 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.426350    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:50.426543 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.426422    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:50.426543 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.426471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:50.426543 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.426513    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:50.426921 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:50.426562    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:50.433915 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.432985    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh" event={"ID":"ee19afb5-85b7-4f2a-b0ae-1a848e32d728","Type":"ContainerStarted","Data":"416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d"}
Oct 30 21:41:50.435250 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.435214    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2gnf" event={"ID":"16ada66f-5869-428a-973b-91d299646056","Type":"ContainerStarted","Data":"a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb"}
Oct 30 21:41:50.436742 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.436721    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe"}
Oct 30 21:41:50.441161 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.441104    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxqp4" event={"ID":"4300224c-3e21-45f0-bf6b-d60a935f3aea","Type":"ContainerStarted","Data":"398aa4afd2430154323ff86a995f1a8b032754505913c6129af72fcdcdc0c0b9"}
Oct 30 21:41:50.442948 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.442926    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fttwq" event={"ID":"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634","Type":"ContainerStarted","Data":"5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09"}
Oct 30 21:41:50.444950 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.444925    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9l2mj" event={"ID":"7524bf5c-af7d-4d94-a660-06fe000da0ae","Type":"ContainerStarted","Data":"c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b"}
Oct 30 21:41:50.447426 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.447399    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n52cs" event={"ID":"1de5e171-5eb9-4628-806b-11b5fc875413","Type":"ContainerStarted","Data":"473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0"}
Oct 30 21:41:50.450968 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.450946    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerStarted","Data":"a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62"}
Oct 30 21:41:50.452576 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.452526    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-st48b" event={"ID":"e7040152-696f-44aa-b210-81c5093eed9d","Type":"ContainerStarted","Data":"7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046"}
Oct 30 21:41:50.543980 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.543935    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:48 +0000 UTC" deadline="2028-04-18 04:17:08.973235962 +0000 UTC"
Oct 30 21:41:50.543980 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:50.543970    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12846h35m18.429269967s"
Oct 30 21:41:52.115074 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.115020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:52.115360 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.115093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:52.115360 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.115227    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:52.115360 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.115285    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:56.115266234 +0000 UTC m=+9.322657689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:52.115360 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.115348    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:52.115502 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.115380    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:56.115369245 +0000 UTC m=+9.322760703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:52.216673 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.216450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:52.216818 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.216630    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:52.216818 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.216750    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:52.216818 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.216764    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:52.216965 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.216905    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:41:56.216886408 +0000 UTC m=+9.424277875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.426015    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.426149    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.426221    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.426289    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.426341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:52.426438 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:52.426404    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:52.476211 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.476175    2584 generic.go:356] "Generic (PLEG): container finished" podID="cf36e468c742c0445d905ae4f02392b0" containerID="86557df9f526551f604ca33b7a33db87c49324288fec37383496796f452af65c" exitCode=0
Oct 30 21:41:52.476391 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.476360    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerDied","Data":"86557df9f526551f604ca33b7a33db87c49324288fec37383496796f452af65c"}
Oct 30 21:41:52.483536 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.483511    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"fe4301f18ed471dd47244e9704246a79","Type":"ContainerStarted","Data":"b8a0841c628b22244a4ba05c4316923e68b820f5dd0513107b0e05523b3cf399"}
Oct 30 21:41:52.724911 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:52.721625    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal" podStartSLOduration=5.72160572 podStartE2EDuration="5.72160572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:52.721527249 +0000 UTC m=+5.928918724" watchObservedRunningTime="2026-10-30 21:41:52.72160572 +0000 UTC m=+5.928997195"
Oct 30 21:41:53.493590 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:53.493516    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerStarted","Data":"0043e74cda51356f9c8a9c7870379b9953f57a87c6264c1507c201ee33c0ff97"}
Oct 30 21:41:53.499489 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:53.499459    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="cffff4cea3f03b0bfe1cdca17acb37fa187d8d60c13c415042d2a0ce00c7edb6" exitCode=0
Oct 30 21:41:53.499620 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:53.499580    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"cffff4cea3f03b0bfe1cdca17acb37fa187d8d60c13c415042d2a0ce00c7edb6"}
Oct 30 21:41:53.745393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:53.745293    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" podStartSLOduration=6.745273594 podStartE2EDuration="6.745273594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:53.61284472 +0000 UTC m=+6.820236197" watchObservedRunningTime="2026-10-30 21:41:53.745273594 +0000 UTC m=+6.952665069"
Oct 30 21:41:54.426796 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.426749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:54.426975 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:54.426890    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:54.426975 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.426964    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:54.427099 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:54.427031    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:54.427099 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.427086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:54.427201 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:54.427164    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:54.514396 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.514358    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9l2mj" event={"ID":"7524bf5c-af7d-4d94-a660-06fe000da0ae","Type":"ContainerStarted","Data":"06e02a2ab432e778e8864d9f3b8d40f7219f8c0da5c3310444852f9f1cdb4c1a"}
Oct 30 21:41:54.539191 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.539089    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:54.540002 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.539960    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:54.742329 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:54.742223    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9l2mj" podStartSLOduration=3.212191436 podStartE2EDuration="7.742210163s" totalImagesPullingTime="4.530018727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.628752263 +0000 UTC m=+7.836143739" watchObservedRunningTime="2026-10-30 21:41:54.742210163 +0000 UTC m=+7.949601638"
Oct 30 21:41:55.519293 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:55.519263    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2gnf" event={"ID":"16ada66f-5869-428a-973b-91d299646056","Type":"ContainerStarted","Data":"815d02ed814ef935bbaaea5c7ad42105b63daedf62c73b1e080ab73c69c761af"}
Oct 30 21:41:55.702690 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:55.702594    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:55.703730 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:55.703529    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:41:55.824609 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:55.824551    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2gnf" podStartSLOduration=3.709265393 podStartE2EDuration="8.824529204s" totalImagesPullingTime="5.115263811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.637157086 +0000 UTC m=+8.844548562" watchObservedRunningTime="2026-10-30 21:41:55.824529204 +0000 UTC m=+9.031920758"
Oct 30 21:41:56.150954 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.150913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:56.151162 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.151002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:56.151162 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.151144    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:56.151277 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.151212    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:04.151193449 +0000 UTC m=+17.358584917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:56.151340 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.151287    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:56.151340 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.151327    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:04.151314562 +0000 UTC m=+17.358706014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:56.251920 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.251879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:56.252099 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.252024    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:56.252099 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.252043    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:56.252099 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.252055    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:56.252257 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.252111    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:42:04.252092957 +0000 UTC m=+17.459484422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:56.426505 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.426150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:56.426505 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.426181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:56.426505 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.426181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:56.426505 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.426281    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:56.426505 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.426393    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:56.426953 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:56.426508    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:56.524824 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:56.524574    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh" event={"ID":"ee19afb5-85b7-4f2a-b0ae-1a848e32d728","Type":"ContainerStarted","Data":"28f61f32711d0ed620fe091f6075620afa204a886f37d7698aedc3736d434421"}
Oct 30 21:41:57.532798 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:57.530023    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fttwq" event={"ID":"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634","Type":"ContainerStarted","Data":"219a373064f01bfc4f4d43f5dcf3e265945b94d7d5a216072a3490901073e0c4"}
Oct 30 21:41:57.663054 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:57.662815    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fttwq" podStartSLOduration=2.7876396100000003 podStartE2EDuration="10.662798318s" totalImagesPullingTime="7.875158708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:57.662207947 +0000 UTC m=+10.869599421" watchObservedRunningTime="2026-10-30 21:41:57.662798318 +0000 UTC m=+10.870189794"
Oct 30 21:41:57.977702 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:57.977681    2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:41:58.240098 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.239948    2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:41:57.977701145Z","UUID":"a425145f-bc7d-4487-bd19-833240d29e40","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:41:58.241524 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.241501    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:41:58.241652 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.241531    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.426887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:58.427024    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.427093    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:58.427147    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.427191    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:41:58.427306 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:41:58.427258    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:41:58.533209 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.533134    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh" event={"ID":"ee19afb5-85b7-4f2a-b0ae-1a848e32d728","Type":"ContainerStarted","Data":"c9ccf51fd3da2b3ff6576f50ebcd9828bbb031b6f50424b921fff973de04d1be"}
Oct 30 21:41:58.535800 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.535399    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="27431a0193aeb2848e655c1642fd3dae3509331cac805225f5400cdbeffc4be7" exitCode=0
Oct 30 21:41:58.535800 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.535472    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"27431a0193aeb2848e655c1642fd3dae3509331cac805225f5400cdbeffc4be7"}
Oct 30 21:41:58.538156 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.537768    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-st48b" event={"ID":"e7040152-696f-44aa-b210-81c5093eed9d","Type":"ContainerStarted","Data":"34af9183f391927e6a67167635abb1643a628ce9d57453f0814e7b489d91ad9f"}
Oct 30 21:41:58.768603 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:58.768553    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-st48b" podStartSLOduration=3.116468499 podStartE2EDuration="11.768540035s" totalImagesPullingTime="8.652071536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:58.76834687 +0000 UTC m=+11.975738346" watchObservedRunningTime="2026-10-30 21:41:58.768540035 +0000 UTC m=+11.975931509"
Oct 30 21:41:59.543485 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:59.543448    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh" event={"ID":"ee19afb5-85b7-4f2a-b0ae-1a848e32d728","Type":"ContainerStarted","Data":"8a8a8857338a06ff76076ed1f375a5357a2a1c6709e9974f5ea99162fc1611fd"}
Oct 30 21:41:59.545119 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:59.545067    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n52cs" event={"ID":"1de5e171-5eb9-4628-806b-11b5fc875413","Type":"ContainerStarted","Data":"7c2200d7079b00565d4efe915ddbdab712a5613780e6ece5e08d5431ccfaaf93"}
Oct 30 21:41:59.659295 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:59.659248    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh" podStartSLOduration=3.377358632 podStartE2EDuration="12.659229909s" totalImagesPullingTime="9.281871277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:59.65893131 +0000 UTC m=+12.866322786" watchObservedRunningTime="2026-10-30 21:41:59.659229909 +0000 UTC m=+12.866621385"
Oct 30 21:41:59.772185 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:41:59.772118    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n52cs" podStartSLOduration=4.62886133 podStartE2EDuration="12.772103094s" totalImagesPullingTime="8.143241764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:59.771545801 +0000 UTC m=+12.978937276" watchObservedRunningTime="2026-10-30 21:41:59.772103094 +0000 UTC m=+12.979494568"
Oct 30 21:42:00.426601 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.426569    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:00.426764 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.426637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:00.426764 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.426742    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:00.426915 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:00.426749    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:00.426915 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:00.426833    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:00.426915 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:00.426900    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:00.549659 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.549625    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="a938f665cc3bbfebdb559e81d0ef7b3b665e170ba6521b6ea7cfd03c293b05cc" exitCode=0
Oct 30 21:42:00.550433 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.549704    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"a938f665cc3bbfebdb559e81d0ef7b3b665e170ba6521b6ea7cfd03c293b05cc"}
Oct 30 21:42:00.551641 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.551356    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxqp4" event={"ID":"4300224c-3e21-45f0-bf6b-d60a935f3aea","Type":"ContainerStarted","Data":"2c558f1819745c31b0ca4e4ed5ab2f0badce2873cff5424307d550209ab40454"}
Oct 30 21:42:00.822247 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:00.822162    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fxqp4" podStartSLOduration=3.451413807 podStartE2EDuration="13.822144149s" totalImagesPullingTime="10.370730342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:00.82149474 +0000 UTC m=+14.028886216" watchObservedRunningTime="2026-10-30 21:42:00.822144149 +0000 UTC m=+14.029535616"
Oct 30 21:42:02.426907 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:02.426705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:02.427398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:02.426705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:02.427398 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:02.426991    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:02.427398 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:02.427096    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:02.427398 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:02.426705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:02.427398 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:02.427230    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:02.561012 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:02.560842    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="3ac6407803f67063341a914dd135f693acd15992a543011126191620b44a5c12" exitCode=0
Oct 30 21:42:02.561012 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:02.560880    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"3ac6407803f67063341a914dd135f693acd15992a543011126191620b44a5c12"}
Oct 30 21:42:04.212680 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.212634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:04.213175 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.212696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:04.213175 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.212809    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:04.213175 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.212879    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.212863784 +0000 UTC m=+33.420255249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:04.213175 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.212814    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:04.213175 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.212966    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret podName:9b6c706e-df72-4ce3-95d7-17f7b670b864 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.2129471 +0000 UTC m=+33.420338572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret") pod "global-pull-secret-syncer-b7t59" (UID: "9b6c706e-df72-4ce3-95d7-17f7b670b864") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:04.313016 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.312974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:04.313164 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.313139    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:04.313164 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.313161    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:04.313250 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.313174    2584 projected.go:196] Error preparing data for projected volume kube-api-access-z244k for pod openshift-network-diagnostics/network-check-target-pjnlr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:04.313250 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.313233    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k podName:842ced52-155b-40f5-9c37-6cac6381cfff nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.313215337 +0000 UTC m=+33.520606794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z244k" (UniqueName: "kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k") pod "network-check-target-pjnlr" (UID: "842ced52-155b-40f5-9c37-6cac6381cfff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:04.426754 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.426721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:04.426754 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.426754    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:04.427035 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.426835    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:04.427035 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:04.426721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:04.427035 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.426953    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:04.427176 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:04.427052    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:05.569339 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569140    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"0d74f4ba4c4143fbe858897cd30176fcd81a2070d8381f85aa019f5b46f23348"}
Oct 30 21:42:05.569841 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569400    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"62323d044452b51ed76c2c1b533424c9abae8c252be60e49596f529eb51599fb"}
Oct 30 21:42:05.569841 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569448    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"6e722a247fb9e38366efeb0fd939191107f9316f5d661a35a104afebdcafba39"}
Oct 30 21:42:05.569841 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569484    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"5b7aec12c3bd5ab6b4c53647c49c350abe69c31c11a8f3a12070d40a9249346c"}
Oct 30 21:42:05.569841 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569516    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"45b8be491efdbd5bd50e248b1e99d75f22f1d1962825dce4f0dac331bfa5a341"}
Oct 30 21:42:05.569841 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.569551    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"595dd57fc7e908d4b4742c46d2dc7a71b7fbe37f3a5247f408200b9852dbba80"}
Oct 30 21:42:05.571974 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.571948    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="068883b2dc6e341054ee4f13045851dec311680daae34b00ee7a7744865826e0" exitCode=0
Oct 30 21:42:05.572093 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:05.571987    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"068883b2dc6e341054ee4f13045851dec311680daae34b00ee7a7744865826e0"}
Oct 30 21:42:06.425899 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:06.425862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:06.426069 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:06.425929    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:06.426069 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:06.426017    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:06.426069 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:06.426022    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:06.426177 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:06.426107    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:06.426221 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:06.426173    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:06.577000 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:06.576971    2584 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="df95a873053b636f86b0d191ce1dfad7cc04593cf402c17dd706e2803926c56b" exitCode=0
Oct 30 21:42:06.577479 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:06.577006    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerDied","Data":"df95a873053b636f86b0d191ce1dfad7cc04593cf402c17dd706e2803926c56b"}
Oct 30 21:42:07.581735 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:07.581639    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" event={"ID":"b96a8d72-5f70-4685-9ade-378699d4d95e","Type":"ContainerStarted","Data":"312cba8839652bc134cf9db7981e1998ae615b5ca45b67d887b01e85c4283696"}
Oct 30 21:42:07.584677 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:07.584654    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"2314811d066729f32642015ddd4b2b0229e73a49607cfaf2fc2ccc2ce095f30d"}
Oct 30 21:42:07.704548 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:07.704507    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vzz5n" podStartSLOduration=8.266989572 podStartE2EDuration="20.704495421s" totalImagesPullingTime="12.437505849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:07.704171433 +0000 UTC m=+20.911562907" watchObservedRunningTime="2026-10-30 21:42:07.704495421 +0000 UTC m=+20.911886894"
Oct 30 21:42:08.426269 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:08.426241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:08.426269 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:08.426257    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:08.426455 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:08.426334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:08.426455 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:08.426347    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:08.426455 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:08.426400    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:08.426536 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:08.426453    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:09.592548 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:09.592237    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" event={"ID":"9222687b-1080-41af-8c30-776403f2c1ec","Type":"ContainerStarted","Data":"8e37cc6029b00f8d3a855fbe974d93f4c5a578211a87921e9fffe3491ea5ea5d"}
Oct 30 21:42:09.593125 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:09.592559    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:09.593125 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:09.592612    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:09.607895 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:09.607872    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:09.890349 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:09.890307    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb" podStartSLOduration=7.749338299 podStartE2EDuration="22.890292968s" totalImagesPullingTime="15.140954669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:09.840566073 +0000 UTC m=+23.047957558" watchObservedRunningTime="2026-10-30 21:42:09.890292968 +0000 UTC m=+23.097684441"
Oct 30 21:42:10.426651 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:10.426621    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:10.426809 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:10.426693    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:10.426809 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:10.426790    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:10.426809 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:10.426791    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:10.426943 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:10.426844    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:10.426981 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:10.426947    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:10.595995 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:10.595965    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:10.611360 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:10.611334    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:11.998147 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:11.998115    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7t59"]
Oct 30 21:42:11.998721 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:11.998703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:11.998857 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:11.998837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:12.001198 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:12.001176    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjnlr"]
Oct 30 21:42:12.001419 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:12.001396    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:12.001539 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:12.001506    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:12.047421 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:12.047400    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2bzm"]
Oct 30 21:42:12.047505 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:12.047472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:12.047548 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:12.047537    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:13.429474 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:13.429455    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:13.429838 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:13.429561    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:13.429838 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:13.429609    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:13.429838 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:13.429688    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:14.426015 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:14.425987    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:14.426201 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:14.426080    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:14.452523 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:14.452489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:14.664752 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:14.664723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:15.426513 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:15.426487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:15.426667 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:15.426594    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:15.426667 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:15.426657    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:15.426804 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:15.426760    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:16.426159 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:16.426132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:16.426523 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:16.426228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t59" podUID="9b6c706e-df72-4ce3-95d7-17f7b670b864"
Oct 30 21:42:16.786145 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:16.786065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.429061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.428070    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:17.429061 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:17.428210    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjnlr" podUID="842ced52-155b-40f5-9c37-6cac6381cfff"
Oct 30 21:42:17.429061 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.428634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:17.429061 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:17.428759    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2bzm" podUID="d31e0d6e-bca7-4d54-87d3-6d94f0473da3"
Oct 30 21:42:17.586748 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.586716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.613550 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.613523    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:42:17.613671 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.613635    2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:42:17.725668 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.725604    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pprh8"]
Oct 30 21:42:17.763190 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.763163    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8zwb"]
Oct 30 21:42:17.763309 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.763274    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.765363 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.765340    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.777103 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.777084    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.810857 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.810830    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pprh8"]
Oct 30 21:42:17.816342 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816317    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.816445 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.816546 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rx5n\" (UniqueName: \"kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.816546 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6g2\" (UniqueName: \"kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.816647 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.816647 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.816637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.818069 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.818047    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8zwb"]
Oct 30 21:42:17.819851 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.819829    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:17.819851 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.819842    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:17.819966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.819891    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\""
Oct 30 21:42:17.819966 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.819945    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:17.828586 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.828568    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:17.828671 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.828598    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:17.828671 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.828616    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\""
Oct 30 21:42:17.917547 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.917624 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx5n\" (UniqueName: \"kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.917624 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6g2\" (UniqueName: \"kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.917624 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.917624 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.917786 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:17.917679    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:17.917786 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917739    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.917875 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:17.917856    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.417815366 +0000 UTC m=+31.625206831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:17.917953 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.917937    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.918198 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.918181    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.921308 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.921290    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:17.974013 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.973994    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rx5n\" (UniqueName: \"kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:17.974281 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:17.974264    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6g2\" (UniqueName: \"kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:18.080100 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.080042    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:18.423166 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.423136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:18.423318 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:18.423246    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:18.423318 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:18.423302    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.423288447 +0000 UTC m=+32.630679903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:18.426218 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.426200    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:18.479180 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.479156    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\""
Oct 30 21:42:18.479460 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.479241    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:18.514743 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.514718    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pprh8"]
Oct 30 21:42:18.517590 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:42:18.517568    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8a4f68_b80e_4050_a188_c9f18c59b89c.slice/crio-3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e WatchSource:0}: Error finding container 3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e: Status 404 returned error can't find the container with id 3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e
Oct 30 21:42:18.613619 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.613596    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pprh8" event={"ID":"ed8a4f68-b80e-4050-a188-c9f18c59b89c","Type":"ContainerStarted","Data":"3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e"}
Oct 30 21:42:18.864764 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.864689    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48tph"]
Oct 30 21:42:18.884739 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.884715    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.925855 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.925824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.925961 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.925884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.926021 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.925974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.926069 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.926037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.926114 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.926068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hm7\" (UniqueName: \"kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:18.927954 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:18.927936    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48tph"]
Oct 30 21:42:19.021268 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.021239    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:19.021268 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.021242    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:19.021268 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.021242    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:19.021487 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.021268    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:19.021487 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.021454    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\""
Oct 30 21:42:19.026685 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026734 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026734 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hm7\" (UniqueName: \"kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026805 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026840 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026876 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.026836    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.026910 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.026874    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.026910 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.026897    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.526882743 +0000 UTC m=+32.734274219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.026985 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.026913    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.026985 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.026931    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.526914555 +0000 UTC m=+32.734306033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.027177 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.027160    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.092796 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.092754    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hm7\" (UniqueName: \"kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.129274 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.129215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.326867 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.326826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.426653 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.426626    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:19.426863 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.426839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:19.430591 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.430564    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:19.430690 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.430662    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.430754 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.430741    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.430719011 +0000 UTC m=+34.638110479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.482275 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.482253    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:19.482545 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.482314    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:19.486799 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.486784    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\""
Oct 30 21:42:19.487335 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.487324    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:19.487335 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.487330    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\""
Oct 30 21:42:19.508596 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.508575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.531553 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.531532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.531653 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.531590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:19.531715 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.531661    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.531715 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.531676    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.531846 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.531726    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.531710648 +0000 UTC m=+33.739102121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.531846 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:19.531739    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.531733145 +0000 UTC m=+33.739124597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.686037 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.685970    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.854303 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:19.854268    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:20.224021 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.223938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:20.236851 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.236825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:20.236997 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.236880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:20.237056 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.237045    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:20.237108 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.237092    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:52.237081219 +0000 UTC m=+65.444472674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : secret "metrics-daemon-secret" not found
Oct 30 21:42:20.239356 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.239331    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:20.268365 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.268345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:20.337804 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.337757    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:20.340570 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.340545    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:20.347281 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.347260    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:20.407477 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.407444    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"]
Oct 30 21:42:20.425801 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.425380    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:20.440031 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.439998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:20.468741 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.468053    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"]
Oct 30 21:42:20.529166 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.529093    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:20.529538 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.529178    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:20.538800 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.538758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:20.538931 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.538834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s824t\" (UniqueName: \"kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t\") pod \"migrator-66df5d496b-z2xdg\" (UID: \"0a94e818-6115-4a17-80c7-0ebe49749ee8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:20.538931 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.538869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:20.539051 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.538984    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:20.539107 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.539045    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:20.539107 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.539056    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.539035283 +0000 UTC m=+35.746426752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:20.539107 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:20.539102    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.539084058 +0000 UTC m=+35.746475538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:20.578619 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.578600    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dzgdj\""
Oct 30 21:42:20.597534 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.597507    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:20.639514 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.639491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s824t\" (UniqueName: \"kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t\") pod \"migrator-66df5d496b-z2xdg\" (UID: \"0a94e818-6115-4a17-80c7-0ebe49749ee8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:20.683980 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.683881    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7t59"]
Oct 30 21:42:20.700133 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.700103    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s824t\" (UniqueName: \"kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t\") pod \"migrator-66df5d496b-z2xdg\" (UID: \"0a94e818-6115-4a17-80c7-0ebe49749ee8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:20.726692 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:42:20.726664    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6c706e_df72_4ce3_95d7_17f7b670b864.slice/crio-0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464 WatchSource:0}: Error finding container 0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464: Status 404 returned error can't find the container with id 0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464
Oct 30 21:42:20.748408 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.748381    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjnlr"]
Oct 30 21:42:20.751393 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.751371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:20.751735 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:42:20.751710    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842ced52_155b_40f5_9c37_6cac6381cfff.slice/crio-87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e WatchSource:0}: Error finding container 87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e: Status 404 returned error can't find the container with id 87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e
Oct 30 21:42:20.775181 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.775155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:20.993921 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:20.993888    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:21.147718 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.147688    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"]
Oct 30 21:42:21.150939 ip-10-0-128-19 kubenswrapper[2584]: W1030 21:42:21.150908    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a94e818_6115_4a17_80c7_0ebe49749ee8.slice/crio-ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970 WatchSource:0}: Error finding container ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970: Status 404 returned error can't find the container with id ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970
Oct 30 21:42:21.181520 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.181488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:21.448337 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.448258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:21.448493 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:21.448420    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:21.448552 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:21.448526    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:25.448471627 +0000 UTC m=+38.655863087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:21.621768 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.621731    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg" event={"ID":"0a94e818-6115-4a17-80c7-0ebe49749ee8","Type":"ContainerStarted","Data":"ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970"}
Oct 30 21:42:21.622886 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.622862    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjnlr" event={"ID":"842ced52-155b-40f5-9c37-6cac6381cfff","Type":"ContainerStarted","Data":"87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e"}
Oct 30 21:42:21.623849 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.623833    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7t59" event={"ID":"9b6c706e-df72-4ce3-95d7-17f7b670b864","Type":"ContainerStarted","Data":"0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464"}
Oct 30 21:42:21.871453 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:21.871313    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.557238 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:22.557192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:22.557406 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:22.557270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:22.557406 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:22.557398    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.557525 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:22.557471    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.557453406 +0000 UTC m=+39.764844877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.558038 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:22.557953    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:22.558038 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:22.558006    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.557991836 +0000 UTC m=+39.765383291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:24.635153 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:24.635115    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pprh8" event={"ID":"ed8a4f68-b80e-4050-a188-c9f18c59b89c","Type":"ContainerStarted","Data":"93c36d9826eb9420adb5e9f513a741fd9eda5755c11d37197ae4127358577511"}
Oct 30 21:42:24.637022 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:24.636997    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg" event={"ID":"0a94e818-6115-4a17-80c7-0ebe49749ee8","Type":"ContainerStarted","Data":"70399b241107ad8100ed6d0f35f0bc0494058c514e40b4f598c49ada952127d2"}
Oct 30 21:42:24.862567 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:24.862513    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pprh8" podStartSLOduration=2.721699656 podStartE2EDuration="7.862499912s" totalImagesPullingTime="5.140800256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.748961685 +0000 UTC m=+37.956353170" watchObservedRunningTime="2026-10-30 21:42:24.862499912 +0000 UTC m=+38.069891389"
Oct 30 21:42:24.862747 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:24.862612    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg" podStartSLOduration=2.3511793069999998 podStartE2EDuration="4.862605895s" totalImagesPullingTime="2.511426588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.861905866 +0000 UTC m=+38.069297373" watchObservedRunningTime="2026-10-30 21:42:24.862605895 +0000 UTC m=+38.069997370"
Oct 30 21:42:25.041588 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:25.041556    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:25.481834 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:25.481796    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:25.482010 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:25.481959    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:25.482069 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:25.482014    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.482001918 +0000 UTC m=+46.689393373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:25.641121 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:25.641081    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjnlr" event={"ID":"842ced52-155b-40f5-9c37-6cac6381cfff","Type":"ContainerStarted","Data":"20674e4a751de9f52c4b05263ba7df26d10395ef7c47fa8c585462f10489ff6d"}
Oct 30 21:42:25.754053 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:25.753712    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pjnlr" podStartSLOduration=34.362242899 podStartE2EDuration="38.753694417s" totalImagesPullingTime="4.391451518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:25.753021645 +0000 UTC m=+38.960413119" watchObservedRunningTime="2026-10-30 21:42:25.753694417 +0000 UTC m=+38.961085894"
Oct 30 21:42:26.589747 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:26.589718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:26.589862 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:26.589817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:26.589899 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:26.589864    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.589941 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:26.589930    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.58991375 +0000 UTC m=+47.797305216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.589941 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:26.589931    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.590012 ip-10-0-128-19 kubenswrapper[2584]: E1030 21:42:26.589968    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.589959791 +0000 UTC m=+47.797351258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.608950 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:26.608923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:26.645944 ip-10-0-128-19 kubenswrapper[2584]: I1030 21:42:26.645920    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:26.830576 ip-10-0-128-19 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:42:26.850125 ip-10-0-128-19 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:42:26.850294 ip-10-0-128-19 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:42:26.850339 ip-10-0-128-19 systemd[1]: kubelet.service: Consumed 2.213s CPU time.
Oct 30 21:42:26.865243 ip-10-0-128-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:42:26.938345 ip-10-0-128-19 kubenswrapper[4410]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.938345 ip-10-0-128-19 kubenswrapper[4410]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:42:26.938995 ip-10-0-128-19 kubenswrapper[4410]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.938995 ip-10-0-128-19 kubenswrapper[4410]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941547    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941618    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941621    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941625    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941628    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941630    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941633    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.941625 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941636    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941639    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941642    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941644    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941647    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941650    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941652    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941655    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941658    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941661    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941663    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941666    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941669    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941673    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941678    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941681    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941684    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941687    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941690    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941693    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.941950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941695    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941698    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941704    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941707    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941710    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941712    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941715    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941717    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941720    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941723    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941725    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941728    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941731    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941734    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941737    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941740    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941743    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941745    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941748    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.942437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941758    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941761    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941764    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941767    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941785    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941791    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941794    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941796    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941799    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941802    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941805    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941807    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941810    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941813    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941815    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941818    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941821    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941824    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941826    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.942956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941829    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941832    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941836    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941839    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941841    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941844    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941846    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941849    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941852    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941854    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941857    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941860    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941862    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941865    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941867    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941870    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941873    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941876    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941879    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941881    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.943437 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941884    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941886    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941889    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941892    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941894    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941897    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941900    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941902    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941905    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941907    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941911    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941914    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941916    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941919    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941922    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941925    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941927    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941930    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941932    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941935    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.943972 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.941938    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942715    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942723    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942727    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942731    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942735    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942738    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942740    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942743    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942746    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942749    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942752    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942757    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942761    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942764    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942767    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942785    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942788    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942791    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.944468 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942794    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942796    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942799    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942802    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942804    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942807    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942810    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942813    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942815    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942818    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942821    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942823    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942826    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942829    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942832    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942834    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942837    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942839    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942842    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942844    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.944950 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942847    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942850    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942852    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942855    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942860    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942863    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942866    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942868    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942871    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942875    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942878    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942880    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942883    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942885    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942888    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942891    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942893    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942896    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942898    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.945431 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942901    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942903    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942906    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942909    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942911    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942914    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942917    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942919    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942922    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942924    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942927    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942930    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942932    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942935    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942937    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942939    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942942    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942947    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942949    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.945953 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942952    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942954    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942957    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942961    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942963    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942966    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942969    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942972    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942975    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942977    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942980    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942983    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942985    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942988    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942990    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942993    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942995    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.942997    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943000    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943002    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.946423 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943005    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943008    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943010    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943013    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943015    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943018    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943021    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943024    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943026    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943029    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943498    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943503    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943506    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943509    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943512    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943514    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943517    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943520    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943523    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943525    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.946926 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943528    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943530    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943533    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943536    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943538    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943541    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943543    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943546    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943549    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943552    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943554    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943557    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943559    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943562    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943565    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943567    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943570    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943572    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943575    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.947440 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943578    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943580    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943583    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943587    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943590    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943594    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943597    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943600    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943602    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943605    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943607    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943610    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943613    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943615    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943619    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943622    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943625    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943628    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943630    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.948121 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943633    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943636    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943638    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943641    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943643    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943646    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943648    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943651    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943654    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943656    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943660    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943664    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943668    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943670    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943673    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943675    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943679    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943682    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943687    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.948613 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943690    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943692    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943695    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943697    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943700    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943703    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943705    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943708    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943710    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943713    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943716    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943719    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943721    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943724    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943726    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943729    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943732    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943734    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943737    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943739    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.949086 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943742    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943745    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943747    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943750    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943752    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943755    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943757    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943760    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943762    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943766    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943783    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943789    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943795    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943799    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943802    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943805    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943807    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943810    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.943812    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943908    4410 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943920    4410 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:42:26.949570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943926    4410 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943930    4410 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943935    4410 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943938    4410 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943943    4410 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943947    4410 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943950    4410 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943954    4410 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943957    4410 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943960    4410 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943964    4410 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943967    4410 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943970    4410 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943973    4410 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943976    4410 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943980    4410 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943985    4410 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943988    4410 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943991    4410 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943994    4410 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.943999    4410 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944003    4410 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944007    4410 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:42:26.950096 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944010    4410 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944014    4410 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944017    4410 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944020    4410 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944023    4410 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944026    4410 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944029    4410 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944034    4410 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944037    4410 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944040    4410 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944043    4410 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944046    4410 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944049    4410 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944052    4410 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944055    4410 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944058    4410 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944062    4410 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944064    4410 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944069    4410 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944072    4410 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944075    4410 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944078    4410 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944081    4410 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944083    4410 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944086    4410 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:42:26.950665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944089    4410 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944092    4410 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944095    4410 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944098    4410 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944104    4410 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944107    4410 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944111    4410 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944114    4410 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944117    4410 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944121    4410 flags.go:64] FLAG: --help="false"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944124    4410 flags.go:64] FLAG: --hostname-override="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944127    4410 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944130    4410 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944133    4410 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944137    4410 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944140    4410 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944143    4410 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944146    4410 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944149    4410 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944152    4410 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944155    4410 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944158    4410 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944161    4410 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944164    4410 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:42:26.951308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944167    4410 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944170    4410 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944172    4410 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944175    4410 flags.go:64] FLAG: --lock-file=""
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944178    4410 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944181    4410 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944184    4410 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944190    4410 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944193    4410 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944196    4410 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944199    4410 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944202    4410 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944207    4410 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944210    4410 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944213    4410 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944217    4410 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944221    4410 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944224    4410 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944228    4410 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944231    4410 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944234    4410 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944237    4410 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944240    4410 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944243    4410 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944246    4410 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:42:26.951911 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944253    4410 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944256    4410 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944259    4410 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944262    4410 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944265    4410 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944268    4410 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944271    4410 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944274    4410 flags.go:64] FLAG: --port="10250"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944277    4410 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944279    4410 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-070da6bf92d26b853"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944283    4410 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944286    4410 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944289    4410 flags.go:64] FLAG: --register-node="true"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944292    4410 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944295    4410 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944298    4410 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944301    4410 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944304    4410 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944308    4410 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944312    4410 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944315    4410 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944318    4410 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944322    4410 flags.go:64] FLAG: --runonce="false"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944325    4410 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944328    4410 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944331    4410 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:42:26.952628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944334    4410 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944337    4410 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944340    4410 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944343    4410 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944347    4410 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944350    4410 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944353    4410 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944356    4410 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944359    4410 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944362    4410 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944365    4410 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944367    4410 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944372    4410 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944376    4410 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944378    4410 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944383    4410 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944386    4410 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944389    4410 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944392    4410 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944397    4410 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944400    4410 flags.go:64] FLAG: --v="2"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944404    4410 flags.go:64] FLAG: --version="false"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944408    4410 flags.go:64] FLAG: --vmodule=""
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944412    4410 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944415    4410 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:42:26.953261 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944537    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944541    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944545    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944547    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944550    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944553    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944556    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944558    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944562    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944565    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944568    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944570    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944573    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944576    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944579    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944581    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944584    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944586    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944589    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.954211 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944593    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944595    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944598    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944601    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944603    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944606    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944608    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944612    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944615    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944618    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944620    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944623    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944625    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944629    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944633    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944637    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944640    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944643    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944645    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.955005 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944648    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944651    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944654    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944657    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944660    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944663    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944665    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944668    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944670    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944673    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944675    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944678    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944681    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944684    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944686    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944689    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944691    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944694    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944696    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944699    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.955614 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944703    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944706    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944708    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944711    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944714    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944716    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944720    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944722    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944725    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944728    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944730    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944733    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944736    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944738    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944741    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944743    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944746    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944749    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944752    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944755    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.956188 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944757    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944760    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944763    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944765    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944768    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944785    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944788    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944791    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944793    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944796    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944799    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944802    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944806    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944809    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944811    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944814    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944817    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944819    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944822    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944824    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.956980 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944827    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944830    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944833    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944837    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944840    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944843    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944846    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944848    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.944874    4410 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944908    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944913    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944917    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944920    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944923    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944926    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.957764 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944929    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944932    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944935    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944937    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944940    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944943    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944945    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944948    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944950    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944953    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944957    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944960    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944963    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944965    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944968    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944971    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944973    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944976    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944978    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944981    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.958364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944983    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944987    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944991    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944995    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.944997    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945000    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945003    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945007    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945010    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945013    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945015    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945018    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945020    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945023    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945026    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945028    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945031    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945033    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945036    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.959127 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945038    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945041    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945044    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945049    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945052    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945054    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945057    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945059    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945062    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945065    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945067    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945070    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945073    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945075    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945078    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945080    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945083    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945086    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945088    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945091    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.959887 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945095    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945099    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945103    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945106    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945109    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945112    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945114    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945118    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945120    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945123    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945126    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945128    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945131    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945134    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945136    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945140    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945142    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945145    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945148    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.960511 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945151    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945153    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945156    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945159    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945161    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945164    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945167    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945169    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945172    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945174    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945177    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945179    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945182    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945186    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945188    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945191    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945195    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945198    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945200    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945203    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.961134 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945205    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.945207    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.948814    4410 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.948829    4410 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.948848    4410 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.948852    4410 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948917    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948923    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948926    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948929    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948932    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948934    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948937    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948940    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948942    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948945    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948948    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948951    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948954    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.961624 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948957    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948959    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948962    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948965    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948968    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948971    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948974    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948976    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948981    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948986    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948990    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948993    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.948996    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949000    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949003    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949005    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949008    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949011    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949013    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949016    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.962114 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949018    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949021    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949023    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949026    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949030    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949033    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949036    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949039    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949042    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949045    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949048    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949051    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949054    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949057    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949059    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949062    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949065    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949068    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949070    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.962596 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949073    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949075    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949079    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949081    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949084    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949087    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949090    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949092    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949095    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949098    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949100    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949103    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949106    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949108    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949111    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949114    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949117    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949120    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949123    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949126    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.963113 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949128    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949131    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949134    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949137    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949139    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949142    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949145    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949148    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949150    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949153    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949156    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949158    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949161    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949163    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949166    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949169    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949172    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949174    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949177    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.963593 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949179    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949182    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949184    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949187    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949189    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949193    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949195    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949198    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949200    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949203    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949206    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949209    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949211    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949214    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949217    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.949243    4410 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.964052 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949378    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949382    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949385    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949388    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949391    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949393    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949396    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949399    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949401    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949404    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949406    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949409    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949411    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949414    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949417    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949420    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949423    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949425    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949428    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949431    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.964461 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949434    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949437    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949439    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949442    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949444    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949447    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949451    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949455    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949459    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949463    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949467    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949470    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949473    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949476    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949479    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949481    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949484    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949486    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.965074 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949489    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949491    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949494    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949496    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949499    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949501    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949504    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949507    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949509    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949512    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949515    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949517    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949520    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949522    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949525    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949527    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949530    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949532    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949535    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949538    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.965524 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949540    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949543    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949546    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949548    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949551    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949554    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949556    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949559    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949562    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949564    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949567    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949569    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949572    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949575    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949578    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949580    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949583    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949586    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949588    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.966038 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949591    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949593    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949597    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949599    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949602    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949604    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949607    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949610    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949612    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949615    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949617    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949620    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949623    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949626    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949629    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949631    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949634    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949636    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949639    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949641    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.966492 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949644    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949647    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949649    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949652    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949654    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949657    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949659    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949662    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949664    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.949687    4410 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949717    4410 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949722    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949724    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949727    4410 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949729    4410 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949732    4410 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.966999 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949735    4410 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949738    4410 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949741    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949745    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949748    4410 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949758    4410 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949762    4410 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949764    4410 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949767    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949784    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949788    4410 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949791    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949793    4410 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949796    4410 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949799    4410 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949801    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949804    4410 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949807    4410 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949809    4410 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.967395 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949813    4410 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949818    4410 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949821    4410 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949826    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949829    4410 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949831    4410 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949834    4410 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949837    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949839    4410 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949842    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949844    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949847    4410 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949849    4410 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949853    4410 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949857    4410 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949860    4410 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949864    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949866    4410 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949869    4410 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.967894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949871    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949874    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949876    4410 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949879    4410 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949882    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949884    4410 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949887    4410 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949889    4410 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949892    4410 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949894    4410 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949897    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949900    4410 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949902    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949905    4410 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949908    4410 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949911    4410 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949913    4410 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949916    4410 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949919    4410 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949921    4410 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.968467 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949924    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949927    4410 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949929    4410 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949932    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949934    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949937    4410 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949939    4410 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949944    4410 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949947    4410 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949951    4410 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949954    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949956    4410 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949959    4410 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949962    4410 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949964    4410 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949967    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949969    4410 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949972    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949974    4410 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949977    4410 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.969262 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949980    4410 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949982    4410 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949985    4410 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949987    4410 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949990    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949993    4410 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949996    4410 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.949998    4410 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950001    4410 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950003    4410 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950006    4410 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950008    4410 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950011    4410 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950013    4410 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950017    4410 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950019    4410 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950022    4410 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950024    4410 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950027    4410 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950031    4410 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.970202 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950034    4410 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:26.950038    4410 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.950085    4410 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951042    4410 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951094    4410 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951261    4410 server.go:1014] "Starting client certificate rotation"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951403    4410 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951454    4410 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:47 +0000 UTC" deadline="2028-07-10 19:54:39.314715136 +0000 UTC"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951498    4410 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14854h12m12.363219904s"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.951783    4410 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.952804    4410 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.955133    4410 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.957258    4410 log.go:25] "Validated CRI v1 image API"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.959515    4410 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:42:26.971093 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.960960    4410 fs.go:135] Filesystem UUIDs: map[6884a548-4cde-47cd-ad52-8041eacac26a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ab474186-8d83-4259-93af-14b0f0d62ae4:/dev/nvme0n1p3]
Oct 30 21:42:26.971758 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.960983    4410 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a94e818-6115-4a17-80c7-0ebe49749ee8/volumes/kubernetes.io~projected/kube-api-access-s824t:{mountpoint:/var/lib/kubelet/pods/0a94e818-6115-4a17-80c7-0ebe49749ee8/volumes/kubernetes.io~projected/kube-api-access-s824t major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16ada66f-5869-428a-973b-91d299646056/volumes/kubernetes.io~projected/kube-api-access-45x8s:{mountpoint:/var/lib/kubelet/pods/16ada66f-5869-428a-973b-91d299646056/volumes/kubernetes.io~projected/kube-api-access-45x8s major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1de5e171-5eb9-4628-806b-11b5fc875413/volumes/kubernetes.io~projected/kube-api-access-pb7sz:{mountpoint:/var/lib/kubelet/pods/1de5e171-5eb9-4628-806b-11b5fc875413/volumes/kubernetes.io~projected/kube-api-access-pb7sz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634/volumes/kubernetes.io~projected/kube-api-access-prd5v:{mountpoint:/var/lib/kubelet/pods/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634/volumes/kubernetes.io~projected/kube-api-access-prd5v major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9e5914-7d34-4602-a1f4-bab25a68879a/volumes/kubernetes.io~projected/kube-api-access-6rx5n:{mountpoint:/var/lib/kubelet/pods/3c9e5914-7d34-4602-a1f4-bab25a68879a/volumes/kubernetes.io~projected/kube-api-access-6rx5n major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~projected/kube-api-access-xjhr7:{mountpoint:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~projected/kube-api-access-xjhr7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7524bf5c-af7d-4d94-a660-06fe000da0ae/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7524bf5c-af7d-4d94-a660-06fe000da0ae/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8105b15f-1a9a-4315-817f-6a6a81cbe206/volumes/kubernetes.io~projected/kube-api-access-q5hm7:{mountpoint:/var/lib/kubelet/pods/8105b15f-1a9a-4315-817f-6a6a81cbe206/volumes/kubernetes.io~projected/kube-api-access-q5hm7 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/842ced52-155b-40f5-9c37-6cac6381cfff/volumes/kubernetes.io~projected/kube-api-access-z244k:{mountpoint:/var/lib/kubelet/pods/842ced52-155b-40f5-9c37-6cac6381cfff/volumes/kubernetes.io~projected/kube-api-access-z244k major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~projected/kube-api-access-9zwxd:{mountpoint:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~projected/kube-api-access-9zwxd major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b6c706e-df72-4ce3-95d7-17f7b670b864/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9b6c706e-df72-4ce3-95d7-17f7b670b864/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96a8d72-5f70-4685-9ade-378699d4d95e/volumes/kubernetes.io~projected/kube-api-access-xvn84:{mountpoint:/var/lib/kubelet/pods/b96a8d72-5f70-4685-9ade-378699d4d95e/volumes/kubernetes.io~projected/kube-api-access-xvn84 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31e0d6e-bca7-4d54-87d3-6d94f0473da3/volumes/kubernetes.io~projected/kube-api-access-jdfls:{mountpoint:/var/lib/kubelet/pods/d31e0d6e-bca7-4d54-87d3-6d94f0473da3/volumes/kubernetes.io~projected/kube-api-access-jdfls major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7040152-696f-44aa-b210-81c5093eed9d/volumes/kubernetes.io~projected/kube-api-access-xlppm:{mountpoint:/var/lib/kubelet/pods/e7040152-696f-44aa-b210-81c5093eed9d/volumes/kubernetes.io~projected/kube-api-access-xlppm major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~projected/kube-api-access-kl6g2:{mountpoint:/var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~projected/kube-api-access-kl6g2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee19afb5-85b7-4f2a-b0ae-1a848e32d728/volumes/kubernetes.io~projected/kube-api-access-txs9p:{mountpoint:/var/lib/kubelet/pods/ee19afb5-85b7-4f2a-b0ae-1a848e32d728/volumes/kubernetes.io~projected/kube-api-access-txs9p major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fba3af4fe05eba17444b639bf4b30443de57dfccba4b3e962c114b3e369b8043/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5594f40f882818f3c6091251b97bf54a09d833d2c61fc66e4a65cf5d9edfcf0d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7dcfbc719bc2c3be2589c97af5be0694820cdaf7a9c2da58c3cd21209e150eda/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f24f977d32f6c16460bbccbd4438d23d485c6e98c64418d3ba83516159bd0a40/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3a8b662c6d4cbe662e5f1bea9e4a33c6f9f863cd0e31cc03041819d59b54083b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f3a18ba200f26810e093a5e63248c5c43d1b68b3eedf3516fb8c9e5a6b55f8dd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a064cb0641d6c75c6df7fac49c559fba989138e9c28f4af921cefbefa39c5505/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d269e0173978d68ec76a1020108a19632d7a8888636fbc363b5a92442f870bd9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e7543fd00cac51fc81085f8425ad923dcec974ab2c6713d55a6051bc5ef91494/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/2545ef196d2abe88197af788d6a04d5a8346e5bbec0df65e5b80ce30f2afc14a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/9a68e93ea3b0b7beaa63c5ff6f8be08cda299da1348f48d5743fec395e0c3632/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/7f442c48b457bcd19ff1507b6c4d715c520d252372028bbb1ce56ee78660f4d2/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fffece94dd18cfd3b0731e8b716e08914748449ace917b23fc1118b296ff687f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d8d564f5c5ab49b3b836117e36bd4d8f94eb14c17753abe4b66288e3af68099f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/bc338d9279370f7070bc53d12d5c823152f176e636f481dfb67d388eb52cdcaa/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/43f53d57aeee21dbe2d5e85f5750a705b2aa0f0f068ea5737005a9b62e57640d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5ab2f54a4d115291d1b35f17e18203bb245e59004b940fffa32e961c8f55dc27/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b77d86c6d37d4d715e753a56b8d3722b17c59b022518df2e692dcf9d5d5e76f7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/451ff7a9c8031944571d493e8069ff0663c900d4d8ec0705af05e36f25831bea/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/029432a33c4364d2b889124a36a4eef89b75503166a44db11d08770725284c8d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/eeea9b0dc703e9c9a7730e4b725c17d5d7929a2f1b8b85125dc60fcc88a26110/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/5948d198605454aee99c05f314c678b27ee28604dad3a6fca3810e3b6b22e3a5/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/2078a7e2200d836c68b72136143761f106bc5d79b58f33e4ac3e734d92dbf262/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/13e8d5749aa1c3b0ee0371bae6423fd256af1d2b77f5d60b47a61bd09e28d0e3/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/1d9aabc0c4b0057009deebfcb81b25e9c3a55727b2ae5c187f3f8c8144b8fc9c/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/a5027971fb93634afd2f7815d530cf2cf56953114796cde1e2fb4ceb3f175206/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/d95a8240e9dc3902c38820d4b1dfbeb2d4762986788bb76ba02e68b3fe9e0149/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/ff3550b36cb46a0cb695bf10ffee6fdbc39aab5d1a8f5ec2bf899e7034ccd6e6/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/c29ec109c7987e7444bd0ace567c1a689532c5fee31c6a7edc4703810294dda6/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/6b32bd69e4451faf943c62239b09583c090c2d5ce0a7202eff50672c34b29ed0/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ee2c2ca3c6a6af38ca13dc7de7d538270b877a2a9157bca82d05d4b5bfcdbd3d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d14ee537a67a2ce541ac4aa98b1c562ce3eb91bcb069f7b4f6aad7340f71b092/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/051b82c28205d295293f2f1d2f1d78267d2fe54945dfe1888d9597a292d76398/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/8824d4c175900bffcdbcaf7a66e1e6d934b5e26f0ca5ba9161744da70502dd19/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d515ce3e993bf82682ea18389ead51e5c0330e47a8ed3c55c4cb30a06589f0c2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a63d2bf6a3d3efb4dcfa699e4a595ad7a5df40434a0f0a24f9d154307e1e2d88/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/fa7f3ba7d9830c4dbebc1f5169677548b6f7952afb565724abe8416a6a754fcf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3dc7c85ecd60e70c4f63b2fbde5a2c905175860171e80b5ae6225585346e4411/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bd548a0413bd9e5490f8e1869c5b3518953a862d30f61187b33835ce8e09a8da/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.966962    4410 manager.go:217] Machine: {Timestamp:2026-10-30 21:42:26.966486946 +0000 UTC m=+0.055224739 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100537 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec267ac6070a302663a68f3eda6ed304 SystemUUID:ec267ac6-070a-3026-63a6-8f3eda6ed304 BootID:2495ca1f-5004-4aa9-8b21-507af91242bc Filesystems:[{Device:/var/lib/kubelet/pods/7524bf5c-af7d-4d94-a660-06fe000da0ae/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ee19afb5-85b7-4f2a-b0ae-1a848e32d728/volumes/kubernetes.io~projected/kube-api-access-txs9p DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec9271922b6c4127eb4db5507b787738959233b6eed17db8a036501571c4c970/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7040152-696f-44aa-b210-81c5093eed9d/volumes/kubernetes.io~projected/kube-api-access-xlppm DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~projected/kube-api-access-xjhr7 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e3ed93754bf4a4d7e98ee01e2387ebafbc70055f3be12320620d8b083410046/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e03081dbadd9c898b53bafb0e88c4d6873a02d3c16aebf5561d4048acdc657e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/842ced52-155b-40f5-9c37-6cac6381cfff/volumes/kubernetes.io~projected/kube-api-access-z244k DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a273be9b5f83927e1cbdda89afc7cf3b8aa28b4218506dc692259056d8d6cb62/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf8e405cf9cf07d253c5f4dad4bd8460fed14cb16a0f675ab2b3f88a860bbebe/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~projected/kube-api-access-9zwxd DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1de5e171-5eb9-4628-806b-11b5fc875413/volumes/kubernetes.io~projected/kube-api-access-pb7sz DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9e5914-7d34-4602-a1f4-bab25a68879a/volumes/kubernetes.io~projected/kube-api-access-6rx5n DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0a94e818-6115-4a17-80c7-0ebe49749ee8/volumes/kubernetes.io~projected/kube-api-access-s824t DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87c5a56099b4efcb07c39df981c5787a0532bc8d57ba8105b4c5f52546558f5e/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8105b15f-1a9a-4315-817f-6a6a81cbe206/volumes/kubernetes.io~projected/kube-api-access-q5hm7 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ba793c38ffa9a0904c54ab1ea8302efb6927abc83e5ab6b4b201d9e33aee464/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9222687b-1080-41af-8c30-776403f2c1ec/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5526ee38f8eae30bfb92b00615e396b2f5266956e594de337750ec962528fc09/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d31e0d6e-bca7-4d54-87d3-6d94f0473da3/volumes/kubernetes.io~projected/kube-api-access-jdfls DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c143feb1503d4f7d823af74405691a7afda7b3bc3a7060e2bc2d39ff2e61ae9b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed8a4f68-b80e-4050-a188-c9f18c59b89c/volumes/kubernetes.io~projected/kube-api-access-kl6g2 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16ada66f-5869-428a-973b-91d299646056/volumes/kubernetes.io~projected/kube-api-access-45x8s DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634/volumes/kubernetes.io~projected/kube-api-access-prd5v DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b96a8d72-5f70-4685-9ade-378699d4d95e/volumes/kubernetes.io~projected/kube-api-access-xvn84 DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9c58dfdb77e5a3346dd7c45251b629b3a599431c0a70df02a0669ffc93c29bb/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4300224c-3e21-45f0-bf6b-d60a935f3aea/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/473c56806538e99542adcbcd6d406d16129919167ae74deeba8ecdbe439843d0/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/416f4e83c35899f0225220029197aa0915a64c07b6772cc7ca52e2dda8230b6d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9b6c706e-df72-4ce3-95d7-17f7b670b864/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ba793c38ffa9a0 MacAddress:12:21:90:28:1d:07 Speed:10000 Mtu:8901} {Name:3e03081dbadd9c8 MacAddress:e2:5d:0e:f1:8f:d7 Speed:10000 Mtu:8901} {Name:87c5a56099b4efc MacAddress:e6:a3:b8:8b:cd:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:19:75:0f:e9:15 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:fd:f9:aa:c1:6b Speed:0 Mtu:8901} {Name:ec9271922b6c412 MacAddress:06:a4:f0:1f:d3:54 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:19:75:0f:e9:15 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a6:a0:e4:5a:09:21 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:57:11:bc:42:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967221    4410 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967264    4410 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967548    4410 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967572    4410 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-19.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967701    4410 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967709    4410 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967720    4410 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967732    4410 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967807    4410 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.967972    4410 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968083    4410 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968117    4410 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968127    4410 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968137    4410 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968148    4410 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968157    4410 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968765    4410 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.968798    4410 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:26.972118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.969256    4410 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:42:26.973192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973176    4410 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:26.973192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973192    4410 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:26.973302 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973292    4410 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:42:26.973487 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973476    4410 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:42:26.973516 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973505    4410 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:42:26.973600 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973591    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:42:26.973628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973605    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:42:26.973628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973611    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:42:26.973628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973616    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:42:26.973628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973622    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:42:26.973628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973627    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973634    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973639    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973646    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973652    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973661    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973670    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973696    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:42:26.973807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.973702    4410 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:42:26.976855 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.976842    4410 server.go:1263] "Started kubelet"
Oct 30 21:42:26.977305 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.976904    4410 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:42:26.977385 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.977245    4410 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:42:26.977385 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.977363    4410 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:42:26.977349 ip-10-0-128-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:42:26.977708 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.977653    4410 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:42:26.978702 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.978687    4410 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:42:26.979565 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.979550    4410 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:42:26.980966 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.980873    4410 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:42:26.980966 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.980889    4410 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:42:26.980966 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:26.980887    4410 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.980989    4410 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982302    4410 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982372    4410 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:48 +0000 UTC" deadline="2028-04-30 10:23:34.129547011 +0000 UTC"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982406    4410 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13140h41m7.147143655s"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982506    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7dc4c7-b8fb-44e8-8934-c44ef5a33634" volumeName="kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982537    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9e5914-7d34-4602-a1f4-bab25a68879a" volumeName="kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982553    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8105b15f-1a9a-4315-817f-6a6a81cbe206" volumeName="kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982565    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed8a4f68-b80e-4050-a188-c9f18c59b89c" volumeName="kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982578    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7dc4c7-b8fb-44e8-8934-c44ef5a33634" volumeName="kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982592    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7524bf5c-af7d-4d94-a660-06fe000da0ae" volumeName="kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982604    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96a8d72-5f70-4685-9ade-378699d4d95e" volumeName="kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982617    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ada66f-5869-428a-973b-91d299646056" volumeName="kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982633    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1de5e171-5eb9-4628-806b-11b5fc875413" volumeName="kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982646    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9e5914-7d34-4602-a1f4-bab25a68879a" volumeName="kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982659    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4300224c-3e21-45f0-bf6b-d60a935f3aea" volumeName="kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982673    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7524bf5c-af7d-4d94-a660-06fe000da0ae" volumeName="kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982686    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9222687b-1080-41af-8c30-776403f2c1ec" volumeName="kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982699    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96a8d72-5f70-4685-9ade-378699d4d95e" volumeName="kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982715    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96a8d72-5f70-4685-9ade-378699d4d95e" volumeName="kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982728    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9222687b-1080-41af-8c30-776403f2c1ec" volumeName="kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982740    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31e0d6e-bca7-4d54-87d3-6d94f0473da3" volumeName="kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982753    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed8a4f68-b80e-4050-a188-c9f18c59b89c" volumeName="kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982764    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1de5e171-5eb9-4628-806b-11b5fc875413" volumeName="kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982785    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9e5914-7d34-4602-a1f4-bab25a68879a" volumeName="kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982793    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9222687b-1080-41af-8c30-776403f2c1ec" volumeName="kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982800    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b6c706e-df72-4ce3-95d7-17f7b670b864" volumeName="kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982810    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7dc4c7-b8fb-44e8-8934-c44ef5a33634" volumeName="kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982818    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="842ced52-155b-40f5-9c37-6cac6381cfff" volumeName="kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982826    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9222687b-1080-41af-8c30-776403f2c1ec" volumeName="kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982834    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a94e818-6115-4a17-80c7-0ebe49749ee8" volumeName="kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982841    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ada66f-5869-428a-973b-91d299646056" volumeName="kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982854    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4300224c-3e21-45f0-bf6b-d60a935f3aea" volumeName="kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982865    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8105b15f-1a9a-4315-817f-6a6a81cbe206" volumeName="kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982872    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9222687b-1080-41af-8c30-776403f2c1ec" volumeName="kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982880    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96a8d72-5f70-4685-9ade-378699d4d95e" volumeName="kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982887    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7040152-696f-44aa-b210-81c5093eed9d" volumeName="kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982894    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7040152-696f-44aa-b210-81c5093eed9d" volumeName="kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982902    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4300224c-3e21-45f0-bf6b-d60a935f3aea" volumeName="kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982909    4410 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee19afb5-85b7-4f2a-b0ae-1a848e32d728" volumeName="kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p" seLinuxMountContext=""
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982916    4410 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:42:26.983902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.982921    4410 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:42:26.986673 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986597    4410 factory.go:55] Registering systemd factory
Oct 30 21:42:26.986673 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986613    4410 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:42:26.986863 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986847    4410 factory.go:153] Registering CRI-O factory
Oct 30 21:42:26.986914 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986867    4410 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:42:26.986966 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986921    4410 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:42:26.986966 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986940    4410 factory.go:103] Registering Raw factory
Oct 30 21:42:26.986966 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.986955    4410 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:42:26.987569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.987554    4410 manager.go:319] Starting recovery of all containers
Oct 30 21:42:26.996943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:26.996743    4410 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:42:27.011692 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.011671    4410 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:42:27.011804 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.011700    4410 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:42:27.011804 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.011719    4410 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:42:27.011804 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.011764    4410 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:42:27.029200 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029172    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="df95a873053b636f86b0d191ce1dfad7cc04593cf402c17dd706e2803926c56b" exitCode=0
Oct 30 21:42:27.029286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029202    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="068883b2dc6e341054ee4f13045851dec311680daae34b00ee7a7744865826e0" exitCode=0
Oct 30 21:42:27.029286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029213    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="3ac6407803f67063341a914dd135f693acd15992a543011126191620b44a5c12" exitCode=0
Oct 30 21:42:27.029286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029221    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="a938f665cc3bbfebdb559e81d0ef7b3b665e170ba6521b6ea7cfd03c293b05cc" exitCode=0
Oct 30 21:42:27.029286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029232    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="27431a0193aeb2848e655c1642fd3dae3509331cac805225f5400cdbeffc4be7" exitCode=0
Oct 30 21:42:27.029286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.029239    4410 generic.go:356] "Generic (PLEG): container finished" podID="b96a8d72-5f70-4685-9ade-378699d4d95e" containerID="cffff4cea3f03b0bfe1cdca17acb37fa187d8d60c13c415042d2a0ce00c7edb6" exitCode=0
Oct 30 21:42:27.045991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.045969    4410 generic.go:356] "Generic (PLEG): container finished" podID="cf36e468c742c0445d905ae4f02392b0" containerID="86557df9f526551f604ca33b7a33db87c49324288fec37383496796f452af65c" exitCode=0
Oct 30 21:42:27.053272 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.053255    4410 manager.go:324] Recovery completed
Oct 30 21:42:27.066764 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.066744    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.067715 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.067699    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.067842 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.067728    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.067842 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.067739    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.068939 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.068925    4410 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:42:27.068979 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.068939    4410 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:42:27.068979 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.068958    4410 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:42:27.069029 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.068941    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.069108 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069092    4410 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:42:27.069141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069111    4410 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:42:27.069141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069135    4410 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:42:27.069198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069144    4410 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:42:27.069198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069151    4410 policy_none.go:50] "Start"
Oct 30 21:42:27.069198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069164    4410 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:42:27.069198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069175    4410 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:42:27.069321 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069306    4410 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:42:27.069321 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069313    4410 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:42:27.069321 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069318    4410 policy_none.go:44] "Start"
Oct 30 21:42:27.069804 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069789    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.069854 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069816    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.069854 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.069830    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.073131 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.073102    4410 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:42:27.076219 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076202    4410 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:42:27.076290 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.076226    4410 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:42:27.076290 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076235    4410 server.go:87] "Starting device plugin registration server"
Oct 30 21:42:27.076537 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076525    4410 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:42:27.076568 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076539    4410 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:42:27.076629 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076614    4410 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:42:27.076745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076713    4410 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:42:27.076816 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076754    4410 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:42:27.076816 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076762    4410 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:42:27.076907 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.076836    4410 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:27.076744276Z","UUID":"9a83370e-9476-41e8-8de4-864fe44185ab","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:42:27.077980 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.077957    4410 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:42:27.078075 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.078016    4410 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-19.us-west-1.compute.internal\" not found"
Oct 30 21:42:27.078148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.078131    4410 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:42:27.078207 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.078153    4410 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:42:27.112763 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.112687    4410 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"]
Oct 30 21:42:27.112856 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.112823    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.113648 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.113633    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.113704 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.113659    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.113704 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.113669    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.113919 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.113908    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.114075 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114062    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.114697 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114668    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.114697 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114694    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.114697 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114676    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.114896 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114708    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.114896 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114730    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.114896 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114743    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.114896 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.114869    4410 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.115059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114968    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"fe4301f18ed471dd47244e9704246a79","Type":"ContainerStarted","Data":"b8a0841c628b22244a4ba05c4316923e68b820f5dd0513107b0e05523b3cf399"}
Oct 30 21:42:27.115059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.114998    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"fe4301f18ed471dd47244e9704246a79","Type":"ContainerStarted","Data":"2bc328364ab00cae578d0dd767b60b0e521b4a97ba26e24ff5d8578971625ff9"}
Oct 30 21:42:27.115059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.115023    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerStarted","Data":"0043e74cda51356f9c8a9c7870379b9953f57a87c6264c1507c201ee33c0ff97"}
Oct 30 21:42:27.115059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.115037    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerDied","Data":"86557df9f526551f604ca33b7a33db87c49324288fec37383496796f452af65c"}
Oct 30 21:42:27.115059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.115048    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal" event={"ID":"cf36e468c742c0445d905ae4f02392b0","Type":"ContainerStarted","Data":"392e9eee19aace0eccbc8f84ec1e4d6ad0e35510b39013c105505c7455d38891"}
Oct 30 21:42:27.115261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.115072    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.117944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.117926    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.118018 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.117960    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.118018 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.117974    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.118109 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.118098    4410 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-19.us-west-1.compute.internal\" not found" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.176260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.176201    4410 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:42:27.177102 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.177087    4410 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:27.177895 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.177877    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:27.177991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.177908    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:27.177991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.177919    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:27.177991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.177953    4410 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.225317 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.225291    4410 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:27.225412 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.225375    4410 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:27.225482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.225463    4410 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:27.225580 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.225559    4410 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:27.277354 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.277328    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.277511 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.277361    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.277511 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.277381    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.281706 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.281686    4410 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.331366 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.331337    4410 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.331488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.331410    4410 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.332066 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.332048    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 21:42:27.332148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.332083    4410 setters.go:546] "Node became not ready" node="ip-10-0-128-19.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:42:27Z","lastTransitionTime":"2026-10-30T21:42:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 21:42:27.340426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.340404    4410 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:27.340476 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.340448    4410 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.340476 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.340460    4410 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.341423 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.341407    4410 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-19.us-west-1.compute.internal" is owned by different node (oldNodeID="e980ead3-4405-44db-b464-cd382b6eba4c", newNodeID=""), cleaning up...
Oct 30 21:42:27.377554 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377506    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.377554 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377532    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.377554 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377550    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.377671 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377589    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe4301f18ed471dd47244e9704246a79-config\") pod \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"fe4301f18ed471dd47244e9704246a79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.377671 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377616    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.377671 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.377616    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf36e468c742c0445d905ae4f02392b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" (UID: \"cf36e468c742c0445d905ae4f02392b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.409846 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.409828    4410 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:27.409913 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:27.409864    4410 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:27.968625 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:27.968597    4410 apiserver.go:52] "Watching apiserver"
Oct 30 21:42:28.048317 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.048293    4410 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:28.048457 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.048384    4410 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:28.071624 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.071602    4410 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:42:28.072035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.072015    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48tph","openshift-multus/multus-fttwq","kube-system/global-pull-secret-syncer-b7t59","kube-system/konnectivity-agent-9l2mj","openshift-network-diagnostics/network-check-target-pjnlr","openshift-image-registry/node-ca-q2gnf","openshift-ovn-kubernetes/ovnkube-node-qltrb","openshift-dns/dns-default-b8zwb","kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-fxqp4","openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg","openshift-network-operator/iptables-alerter-n52cs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-vzz5n","openshift-ingress-canary/ingress-canary-pprh8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh","openshift-multus/network-metrics-daemon-q2bzm","openshift-dns/node-resolver-st48b"]
Oct 30 21:42:28.077115 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.077076    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.078865 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.078849    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.079317 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.079301    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.082764 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.082746    4410 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:42:28.111867 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.111849    4410 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:28.111934 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.111884    4410 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:28.113736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.113724    4410 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:28.113809 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.113758    4410 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-19.us-west-1.compute.internal"
Oct 30 21:42:28.125402 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125381    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:42:28.125639 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125619    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.125735 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125707    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:42:28.125820 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125740    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\""
Oct 30 21:42:28.125820 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125758    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:42:28.125820 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125808    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125829    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125833    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125869    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125882    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125887    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125900    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125932    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125939    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:42:28.125971 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.125960    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126004    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126016    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126043    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126050    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126097    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126097    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126109    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126108    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126097    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:28.126319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126213    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.126645 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126388    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:42:28.126645 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126525    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\""
Oct 30 21:42:28.126645 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.126607    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\""
Oct 30 21:42:28.128290 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.128276    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:42:28.153259 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153237    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.153259 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153249    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.153387 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153371    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.153442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153398    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.153442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153427    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.153509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153451    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:42:28.153509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.153458    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.183023 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183002    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.183100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183026    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.183100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183043    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.183100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183057    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.183100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183074    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183121    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183164    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183204    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183221    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183285    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183299    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-tuned\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.183356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183334    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:28.183664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183496    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.183664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183560    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45x8s\" (UniqueName: \"kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.183664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183612    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.183814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183713    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7sz\" (UniqueName: \"kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.183814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183798    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.183921 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183892    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.183970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183928    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184006 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.183984    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184005    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.184052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184023    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.184052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184042    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.184181 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184072    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184181 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184106    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184181 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184126    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.184181 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184163    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184186    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184217    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184239    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184280    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hm7\" (UniqueName: \"kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184297    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4300224c-3e21-45f0-bf6b-d60a935f3aea-tmp\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184331    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184347    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184367    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd5v\" (UniqueName: \"kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184441    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-daemon-config\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184464    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184522    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.184588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184537    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184587    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184632    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184666    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184682    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e5914-7d34-4602-a1f4-bab25a68879a-tmp-dir\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184685    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:28.184750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184716    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184750    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184768    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184876    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184901    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184922    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184942    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.184968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184967    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184986    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.184995    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ada66f-5869-428a-973b-91d299646056-serviceca\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185000    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185023    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185052    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b6c706e-df72-4ce3-95d7-17f7b670b864-original-pull-secret\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185085    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185112    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6g2\" (UniqueName: \"kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185126    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-env-overrides\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185139    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7040152-696f-44aa-b210-81c5093eed9d-tmp-dir\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185150    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185168    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvn84\" (UniqueName: \"kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185190    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhr7\" (UniqueName: \"kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.185277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185245    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1de5e171-5eb9-4628-806b-11b5fc875413-iptables-alerter-script\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185296    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx5n\" (UniqueName: \"kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185314    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cni-binary-copy\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185328    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185355    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185412    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185410    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185440    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185461    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185477    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185491    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185520    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185540    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185558    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185575    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185599    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185619    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185633    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185648    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185667    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185702    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185748    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7524bf5c-af7d-4d94-a660-06fe000da0ae-konnectivity-ca\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185765    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185797    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185795    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9222687b-1080-41af-8c30-776403f2c1ec-ovn-node-metrics-cert\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185867    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185883    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-config\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.185868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185889    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txs9p\" (UniqueName: \"kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185898    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b96a8d72-5f70-4685-9ade-378699d4d95e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185918    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185952    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185978    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8105b15f-1a9a-4315-817f-6a6a81cbe206-data-volume\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.185995    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfls\" (UniqueName: \"kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186022    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186119    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9222687b-1080-41af-8c30-776403f2c1ec-ovnkube-script-lib\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186130    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186224    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186278    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186303    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186333    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7524bf5c-af7d-4d94-a660-06fe000da0ae-agent-certs\") pod \"konnectivity-agent-9l2mj\" (UID: \"7524bf5c-af7d-4d94-a660-06fe000da0ae\") " pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186358    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186404    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186442    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s824t\" (UniqueName: \"kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t\") pod \"migrator-66df5d496b-z2xdg\" (UID: \"0a94e818-6115-4a17-80c7-0ebe49749ee8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186475    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186492    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwxd\" (UniqueName: \"kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186513    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlppm\" (UniqueName: \"kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.186936 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.186528    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.225323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.225264    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:28.225323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.225291    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.225949 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.225917    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.226085 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.225972    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:28.226085 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.225990    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:28.226085 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226072    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\""
Oct 30 21:42:28.226230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226122    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:28.226230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226146    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.226230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226148    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\""
Oct 30 21:42:28.226230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226197    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:42:28.226230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226206    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226224    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226230    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226247    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226290    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226331    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dzgdj\""
Oct 30 21:42:28.226431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226387    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.226608 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226472    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\""
Oct 30 21:42:28.226608 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226542    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\""
Oct 30 21:42:28.226690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226611    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\""
Oct 30 21:42:28.226690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226665    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\""
Oct 30 21:42:28.226797 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.226721    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:42:28.229248 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.229228    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.235030 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.235006    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8a4f68-b80e-4050-a188-c9f18c59b89c-cert\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:28.236156 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.236141    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9e5914-7d34-4602-a1f4-bab25a68879a-config-volume\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.242568 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.242544    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7sz\" (UniqueName: \"kubernetes.io/projected/1de5e171-5eb9-4628-806b-11b5fc875413-kube-api-access-pb7sz\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.243165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.243135    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hm7\" (UniqueName: \"kubernetes.io/projected/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-api-access-q5hm7\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.243239 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.243127    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rx5n\" (UniqueName: \"kubernetes.io/projected/3c9e5914-7d34-4602-a1f4-bab25a68879a-kube-api-access-6rx5n\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.243373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.243354    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45x8s\" (UniqueName: \"kubernetes.io/projected/16ada66f-5869-428a-973b-91d299646056-kube-api-access-45x8s\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.244006 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.243981    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhr7\" (UniqueName: \"kubernetes.io/projected/4300224c-3e21-45f0-bf6b-d60a935f3aea-kube-api-access-xjhr7\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.244817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.244339    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvn84\" (UniqueName: \"kubernetes.io/projected/b96a8d72-5f70-4685-9ade-378699d4d95e-kube-api-access-xvn84\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.244817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.244368    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z244k\" (UniqueName: \"kubernetes.io/projected/842ced52-155b-40f5-9c37-6cac6381cfff-kube-api-access-z244k\") pod \"network-check-target-pjnlr\" (UID: \"842ced52-155b-40f5-9c37-6cac6381cfff\") " pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:28.244817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.244513    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd5v\" (UniqueName: \"kubernetes.io/projected/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-kube-api-access-prd5v\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.244996 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.244887    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfls\" (UniqueName: \"kubernetes.io/projected/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-kube-api-access-jdfls\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.244996 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.244897    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwxd\" (UniqueName: \"kubernetes.io/projected/9222687b-1080-41af-8c30-776403f2c1ec-kube-api-access-9zwxd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.245083 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.245064    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6g2\" (UniqueName: \"kubernetes.io/projected/ed8a4f68-b80e-4050-a188-c9f18c59b89c-kube-api-access-kl6g2\") pod \"ingress-canary-pprh8\" (UID: \"ed8a4f68-b80e-4050-a188-c9f18c59b89c\") " pod="openshift-ingress-canary/ingress-canary-pprh8"
Oct 30 21:42:28.245138 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.245083    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlppm\" (UniqueName: \"kubernetes.io/projected/e7040152-696f-44aa-b210-81c5093eed9d-kube-api-access-xlppm\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.245241 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.245220    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txs9p\" (UniqueName: \"kubernetes.io/projected/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kube-api-access-txs9p\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.245514 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.245499    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s824t\" (UniqueName: \"kubernetes.io/projected/0a94e818-6115-4a17-80c7-0ebe49749ee8-kube-api-access-s824t\") pod \"migrator-66df5d496b-z2xdg\" (UID: \"0a94e818-6115-4a17-80c7-0ebe49749ee8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2xdg"
Oct 30 21:42:28.287530 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287509    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.287627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287536    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287572    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.287627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287609    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287621    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-sys-fs\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287630    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-system-cni-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287643    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287662    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-run\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287694    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-host\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287700    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287724    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287735    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287747    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysconfig\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287754    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-conf\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287850    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287877    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-kubelet-config\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.287886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287879    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287851    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-lib-modules\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287907    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287912    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-systemd-units\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287932    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-var-lib-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287949    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287972    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.287987    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288002    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288003    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-socket-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288017    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288034    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288039    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-etc-openvswitch\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288044    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-ovn\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288051    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288054    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288082    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-slash\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288089    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-k8s-cni-cncf-io\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288096    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b6c706e-df72-4ce3-95d7-17f7b670b864-dbus\") pod \"global-pull-secret-syncer-b7t59\" (UID: \"9b6c706e-df72-4ce3-95d7-17f7b670b864\") " pod="kube-system/global-pull-secret-syncer-b7t59"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288123    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288151    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288180    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288233    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-netns\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288243    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288281    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288307    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288330    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288341    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-var-lib-kubelet\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288310    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-etc-kubernetes\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288352    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288380    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-registration-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288387    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-sysctl-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288399    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-log-socket\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.288418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288414    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288447    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288477    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-system-cni-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288484    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-hostroot\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288470    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288580    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288596    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288617    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288636    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-modprobe-d\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288648    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-multus\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288651    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288645    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-sys\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288650    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-systemd\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288677    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ada66f-5869-428a-973b-91d299646056-host\") pod \"node-ca-q2gnf\" (UID: \"16ada66f-5869-428a-973b-91d299646056\") " pod="openshift-image-registry/node-ca-q2gnf"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288682    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288705    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-run-systemd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288710    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288708    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288733    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-node-log\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288740    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288761    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288795    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288800    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288819    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288833    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-netd\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288843    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288847    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-socket-dir-parent\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288856    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-cni-bin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288875    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288885    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-var-lib-kubelet\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288908    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-cnibin\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288901    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288932    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288945    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288970    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.288993    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4300224c-3e21-45f0-bf6b-d60a935f3aea-etc-kubernetes\") pod \"tuned-fxqp4\" (UID: \"4300224c-3e21-45f0-bf6b-d60a935f3aea\") " pod="openshift-cluster-node-tuning-operator/tuned-fxqp4"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289002    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8105b15f-1a9a-4315-817f-6a6a81cbe206-crio-socket\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289003    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289052    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-cni-bin\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289081    4410 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289086    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-netns\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289088    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289163    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.789144692 +0000 UTC m=+1.877882486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : secret "metrics-daemon-secret" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289176    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289194    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289212    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289177    4410 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289229    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de5e171-5eb9-4628-806b-11b5fc875413-host-slash\") pod \"iptables-alerter-n52cs\" (UID: \"1de5e171-5eb9-4628-806b-11b5fc875413\") " pod="openshift-network-operator/iptables-alerter-n52cs"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289237    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289266    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289274    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee19afb5-85b7-4f2a-b0ae-1a848e32d728-device-dir\") pod \"aws-ebs-csi-driver-node-hkdjh\" (UID: \"ee19afb5-85b7-4f2a-b0ae-1a848e32d728\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkdjh"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289295    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289282    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b96a8d72-5f70-4685-9ade-378699d4d95e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzz5n\" (UID: \"b96a8d72-5f70-4685-9ade-378699d4d95e\") " pod="openshift-multus/multus-additional-cni-plugins-vzz5n"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289326    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289341    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-kubelet\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-cnibin\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289369    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289374    4410 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289388    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.789372401 +0000 UTC m=+1.878110196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289395    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9222687b-1080-41af-8c30-776403f2c1ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltrb\" (UID: \"9222687b-1080-41af-8c30-776403f2c1ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289422    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.789408991 +0000 UTC m=+1.878146800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289439    4410 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289495    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289437    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7040152-696f-44aa-b210-81c5093eed9d-hosts-file\") pod \"node-resolver-st48b\" (UID: \"e7040152-696f-44aa-b210-81c5093eed9d\") " pod="openshift-dns/node-resolver-st48b"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.289546    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.789530069 +0000 UTC m=+1.878267865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289573    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289597    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289621    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-multus-conf-dir\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.289650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.289650    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b7dc4c7-b8fb-44e8-8934-c44ef5a33634-host-run-multus-certs\") pod \"multus-fttwq\" (UID: \"3b7dc4c7-b8fb-44e8-8934-c44ef5a33634\") " pod="openshift-multus/multus-fttwq"
Oct 30 21:42:28.412989 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.412957    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.492120 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.492043    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pprh8_ed8a4f68-b80e-4050-a188-c9f18c59b89c/serve-healthcheck-canary/0.log"
Oct 30 21:42:28.672106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.672074    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.730285 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.730260    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:28.792541 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.792484    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:28.792541 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.792511    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:28.792541 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.792530    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.792551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792617    4410 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792630    4410 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792640    4410 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792675    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.792662099 +0000 UTC m=+2.881399883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : secret "metrics-daemon-secret" not found
Oct 30 21:42:28.792707 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792701    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.792685587 +0000 UTC m=+2.881423372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.792900 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792722    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.79270865 +0000 UTC m=+2.881446460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.792900 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792638    4410 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.792900 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:28.792789    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.792762518 +0000 UTC m=+2.881500304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.877169 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:28.877141    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.063108 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.063035    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.093828 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.093798    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-z2xdg_0a94e818-6115-4a17-80c7-0ebe49749ee8/migrator/0.log"
Oct 30 21:42:29.477549 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.477517    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.808275 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.808201    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:29.808275 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.808238    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:29.808275 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.808268    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:29.808293    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808374    4410 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808391    4410 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808406    4410 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808436    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls podName:3c9e5914-7d34-4602-a1f4-bab25a68879a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.80842101 +0000 UTC m=+4.897158790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls") pod "dns-default-b8zwb" (UID: "3c9e5914-7d34-4602-a1f4-bab25a68879a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808456    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.808444374 +0000 UTC m=+4.897182155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808472    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.808463099 +0000 UTC m=+4.897200896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:29.808478 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808482    4410 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:29.808744 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:29.808508    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs podName:d31e0d6e-bca7-4d54-87d3-6d94f0473da3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.808502164 +0000 UTC m=+4.897239944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs") pod "network-metrics-daemon-q2bzm" (UID: "d31e0d6e-bca7-4d54-87d3-6d94f0473da3") : secret "metrics-daemon-secret" not found
Oct 30 21:42:30.271063 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:30.271033    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.686489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:30.686456    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.866242 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:30.866210    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.071434 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.071343    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.451514 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.451480    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.820762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.820684    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:31.820762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.820719    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:31.820762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.820740    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:31.820762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.820759    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:31.821033 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:31.820866    4410 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:31.821033 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:31.820887    4410 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.821033 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:31.820934    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.820919412 +0000 UTC m=+8.909657192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:31.821033 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:31.820950    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.820943752 +0000 UTC m=+8.909681568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.821178 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.821144    4410 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:42:31.823185 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.823164    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d31e0d6e-bca7-4d54-87d3-6d94f0473da3-metrics-certs\") pod \"network-metrics-daemon-q2bzm\" (UID: \"d31e0d6e-bca7-4d54-87d3-6d94f0473da3\") " pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:31.823300 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.823258    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9e5914-7d34-4602-a1f4-bab25a68879a-metrics-tls\") pod \"dns-default-b8zwb\" (UID: \"3c9e5914-7d34-4602-a1f4-bab25a68879a\") " pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:31.862529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.862510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.978562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.978539    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2bzm"
Oct 30 21:42:31.979683 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:31.979643    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:32.065615 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.065584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.236473 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.236446    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.411130 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.411093    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.441069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.440999    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8zwb"]
Oct 30 21:42:32.444364 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:32.444326    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9e5914_7d34_4602_a1f4_bab25a68879a.slice/crio-f8d238367649e0bdb8d47327181fb52220b5dd06155f440b702e11c3a3d086f6 WatchSource:0}: Error finding container f8d238367649e0bdb8d47327181fb52220b5dd06155f440b702e11c3a3d086f6: Status 404 returned error can't find the container with id f8d238367649e0bdb8d47327181fb52220b5dd06155f440b702e11c3a3d086f6
Oct 30 21:42:32.446391 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.446376    4410 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:42:32.490436 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.490390    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2bzm"]
Oct 30 21:42:32.493402 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:32.493378    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31e0d6e_bca7_4d54_87d3_6d94f0473da3.slice/crio-ba3ab8fa9c7da64ccd003568bc2a469e3e294072c73da0a3cf3e62c2b30f8b3b WatchSource:0}: Error finding container ba3ab8fa9c7da64ccd003568bc2a469e3e294072c73da0a3cf3e62c2b30f8b3b: Status 404 returned error can't find the container with id ba3ab8fa9c7da64ccd003568bc2a469e3e294072c73da0a3cf3e62c2b30f8b3b
Oct 30 21:42:32.671479 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.671454    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.902905 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:32.902875    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.058993 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.058945    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8zwb" event={"ID":"3c9e5914-7d34-4602-a1f4-bab25a68879a","Type":"ContainerStarted","Data":"f8d238367649e0bdb8d47327181fb52220b5dd06155f440b702e11c3a3d086f6"}
Oct 30 21:42:33.059965 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.059944    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2bzm" event={"ID":"d31e0d6e-bca7-4d54-87d3-6d94f0473da3","Type":"ContainerStarted","Data":"ba3ab8fa9c7da64ccd003568bc2a469e3e294072c73da0a3cf3e62c2b30f8b3b"}
Oct 30 21:42:33.067626 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.067600    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.226647 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.226584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.389991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.389950    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.672824 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:33.672795    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:34.067992 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:34.067968    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2bzm" event={"ID":"d31e0d6e-bca7-4d54-87d3-6d94f0473da3","Type":"ContainerStarted","Data":"c65a68b7dec62eb822a2d938cfe80386f501ce1d60b79993cb5755d557a792fa"}
Oct 30 21:42:34.477438 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:34.477409    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.664557 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:34.664526    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.889360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:34.889332    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.072916 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.072869    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8zwb" event={"ID":"3c9e5914-7d34-4602-a1f4-bab25a68879a","Type":"ContainerStarted","Data":"f940c47f6b6bcbc1cb856f0f26ab357d21e1d84004984284c9d42acbaf9750dc"}
Oct 30 21:42:35.073072 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.072938    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8zwb" event={"ID":"3c9e5914-7d34-4602-a1f4-bab25a68879a","Type":"ContainerStarted","Data":"0eaca019d46c3e0b68006963aa378f1999d9889860fba239ed3efffa93bfd3f3"}
Oct 30 21:42:35.073072 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.072969    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:35.074742 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.074715    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2bzm" event={"ID":"d31e0d6e-bca7-4d54-87d3-6d94f0473da3","Type":"ContainerStarted","Data":"0bad38de05b831c0c6de1f0cd16cf95ca8379975302ba9d99ace8ddcebac5cfb"}
Oct 30 21:42:35.075250 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.075227    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.306422 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.306337    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.556260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.556234    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.845915 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.845877    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:35.845915 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.845917    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:35.846141 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:35.846009    4410 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:35.846141 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:35.846068    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls podName:8105b15f-1a9a-4315-817f-6a6a81cbe206 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:43.846055084 +0000 UTC m=+16.934792868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48tph" (UID: "8105b15f-1a9a-4315-817f-6a6a81cbe206") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:35.846370 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:35.846351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8105b15f-1a9a-4315-817f-6a6a81cbe206-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:36.271184 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.271149    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.477165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.477130    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.633460 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.633381    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.796233 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.796198    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.944813 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.944766    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:36.944989 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.944978    4410 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:36.956493 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:36.956465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:37.106748 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:37.106720    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:37.461889 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:37.461863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:37.506309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:37.506284    4410 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-19.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:42:37.875178 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:37.875105    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:38.057251 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:38.057220    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:38.333420 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:38.333397    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9l2mj"
Oct 30 21:42:38.679536 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:38.679500    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:38.879276 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:38.879242    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:40.415186 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.415156    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"]
Oct 30 21:42:40.417085 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.417057    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:40.463657 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.463633    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-tgm2b"]
Oct 30 21:42:40.467917 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.467895    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.476525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.476504    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412548c7-be96-4d51-810c-70dd2271de99-tmp\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.476618 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.476549    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qprwm\" (UID: \"1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:40.476663 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.476638    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq57\" (UniqueName: \"kubernetes.io/projected/412548c7-be96-4d51-810c-70dd2271de99-kube-api-access-8nq57\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.512156 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.512138    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:42:40.512470 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.512450    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mqhdp\""
Oct 30 21:42:40.514888 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.514864    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"]
Oct 30 21:42:40.521101 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.521084    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:42:40.521101 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.521088    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:42:40.528497 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.528459    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:40.528621 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.528606    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tgm2b"]
Oct 30 21:42:40.562363 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.562341    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66684b567d-8qpz4"]
Oct 30 21:42:40.563929 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.563913    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.569608 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.569507    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-8zqjj\""
Oct 30 21:42:40.577682 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577661    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq57\" (UniqueName: \"kubernetes.io/projected/412548c7-be96-4d51-810c-70dd2271de99-kube-api-access-8nq57\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.577763 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577701    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8r6\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-kube-api-access-4f8r6\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.577763 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577724    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-trusted-ca\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.577763 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577749    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412548c7-be96-4d51-810c-70dd2271de99-tmp\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.577937 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577825    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-image-registry-private-configuration\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.577937 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577892    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-tls\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.577937 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577918    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-installation-pull-secrets\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.577937 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577937    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qprwm\" (UID: \"1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:40.578143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.577955    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5206b9ba-760c-48f5-9f5b-859f8d5a2345-ca-trust-extracted\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.578195 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.578144    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-certificates\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.578195 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.578177    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-bound-sa-token\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.578291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.578195    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412548c7-be96-4d51-810c-70dd2271de99-tmp\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.580196 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.580177    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qprwm\" (UID: \"1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:40.609981 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.609962    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66684b567d-8qpz4"]
Oct 30 21:42:40.666392 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.666295    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:42:40.668863 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.668830    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq57\" (UniqueName: \"kubernetes.io/projected/412548c7-be96-4d51-810c-70dd2271de99-kube-api-access-8nq57\") pod \"downloads-6dd789c8b7-tgm2b\" (UID: \"412548c7-be96-4d51-810c-70dd2271de99\") " pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.670199 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.670173    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:42:40.670310 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.670211    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:42:40.670369 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.670325    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:42:40.670422 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.670409    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7fm9b\""
Oct 30 21:42:40.678630 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678612    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8r6\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-kube-api-access-4f8r6\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678748 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678638    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-trusted-ca\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678748 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678660    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-image-registry-private-configuration\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678748 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678690    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-tls\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678874 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678819    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-installation-pull-secrets\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678874 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678866    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5206b9ba-760c-48f5-9f5b-859f8d5a2345-ca-trust-extracted\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678955 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678915    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-certificates\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.678955 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.678941    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-bound-sa-token\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.679377 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.679351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5206b9ba-760c-48f5-9f5b-859f8d5a2345-ca-trust-extracted\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.679849 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.679824    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-certificates\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.679849 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.679839    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5206b9ba-760c-48f5-9f5b-859f8d5a2345-trusted-ca\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.681103 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.681076    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-installation-pull-secrets\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.681188 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.681117    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5206b9ba-760c-48f5-9f5b-859f8d5a2345-image-registry-private-configuration\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.681256 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.681241    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-registry-tls\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.729406 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.729384    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:40.735459 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.735430    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8r6\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-kube-api-access-4f8r6\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.735533 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.735507    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5206b9ba-760c-48f5-9f5b-859f8d5a2345-bound-sa-token\") pod \"image-registry-66684b567d-8qpz4\" (UID: \"5206b9ba-760c-48f5-9f5b-859f8d5a2345\") " pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.742451 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.742422    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:40.778625 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.777105    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:42:40.873135 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.873103    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:40.900378 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:40.900327    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.076218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:41.076186    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.091970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:41.091947    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"]
Oct 30 21:42:41.095292 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:41.095264    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da1b6c5_5490_4b39_a6c5_c605cb9e6cc2.slice/crio-e348f9ae0099d464a405eec841c13f52be45587b21bbde23cefea1e42d411735 WatchSource:0}: Error finding container e348f9ae0099d464a405eec841c13f52be45587b21bbde23cefea1e42d411735: Status 404 returned error can't find the container with id e348f9ae0099d464a405eec841c13f52be45587b21bbde23cefea1e42d411735
Oct 30 21:42:41.142118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:41.142095    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tgm2b"]
Oct 30 21:42:41.145401 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:41.145380    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412548c7_be96_4d51_810c_70dd2271de99.slice/crio-48230ac653349cad0ab02115f1735cd9152194501a711fb91f056e17ed5cbdf6 WatchSource:0}: Error finding container 48230ac653349cad0ab02115f1735cd9152194501a711fb91f056e17ed5cbdf6: Status 404 returned error can't find the container with id 48230ac653349cad0ab02115f1735cd9152194501a711fb91f056e17ed5cbdf6
Oct 30 21:42:41.238938 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:41.238746    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66684b567d-8qpz4"]
Oct 30 21:42:41.241311 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:41.241287    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5206b9ba_760c_48f5_9f5b_859f8d5a2345.slice/crio-d80350d27de9fa9797e4ee71a82026200bd185ba1680622e56a5bdbaf01246a2 WatchSource:0}: Error finding container d80350d27de9fa9797e4ee71a82026200bd185ba1680622e56a5bdbaf01246a2: Status 404 returned error can't find the container with id d80350d27de9fa9797e4ee71a82026200bd185ba1680622e56a5bdbaf01246a2
Oct 30 21:42:41.243628 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:41.243600    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.104974 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:42.101362    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tgm2b" event={"ID":"412548c7-be96-4d51-810c-70dd2271de99","Type":"ContainerStarted","Data":"48230ac653349cad0ab02115f1735cd9152194501a711fb91f056e17ed5cbdf6"}
Oct 30 21:42:42.105859 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:42.105815    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm" event={"ID":"1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2","Type":"ContainerStarted","Data":"e348f9ae0099d464a405eec841c13f52be45587b21bbde23cefea1e42d411735"}
Oct 30 21:42:42.107497 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:42.107475    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66684b567d-8qpz4" event={"ID":"5206b9ba-760c-48f5-9f5b-859f8d5a2345","Type":"ContainerStarted","Data":"3a97294956c02872e5bd7d9ee9bba4cbc624ba74f07e6d0d2b5821463f549797"}
Oct 30 21:42:42.107627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:42.107514    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66684b567d-8qpz4" event={"ID":"5206b9ba-760c-48f5-9f5b-859f8d5a2345","Type":"ContainerStarted","Data":"d80350d27de9fa9797e4ee71a82026200bd185ba1680622e56a5bdbaf01246a2"}
Oct 30 21:42:42.107722 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:42.107705    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:42:43.113344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.113213    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm" event={"ID":"1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2","Type":"ContainerStarted","Data":"714466c4befb350c8b5b3ae4b9691f355f8abe499fb86d92c8f3675cb2d86032"}
Oct 30 21:42:43.113765 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.113493    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:43.118583 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.118561    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm"
Oct 30 21:42:43.226013 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.225958    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66684b567d-8qpz4" podStartSLOduration=3.225939178 podStartE2EDuration="3.225939178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:42.228243983 +0000 UTC m=+15.316981785" watchObservedRunningTime="2026-10-30 21:42:43.225939178 +0000 UTC m=+16.314676983"
Oct 30 21:42:43.226397 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.226354    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qprwm" podStartSLOduration=1.571371623 podStartE2EDuration="3.226344654s" totalImagesPullingTime="1.654973031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:43.225867572 +0000 UTC m=+16.314605376" watchObservedRunningTime="2026-10-30 21:42:43.226344654 +0000 UTC m=+16.315082460"
Oct 30 21:42:43.906681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.906646    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:43.909136 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.909106    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8105b15f-1a9a-4315-817f-6a6a81cbe206-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48tph\" (UID: \"8105b15f-1a9a-4315-817f-6a6a81cbe206\") " pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:43.980069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:43.980040    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48tph"
Oct 30 21:42:44.088621 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.088583    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.241476 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.241401    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.398558 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.398525    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.401374 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.401344    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48tph"]
Oct 30 21:42:44.406332 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:44.406305    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8105b15f_1a9a_4315_817f_6a6a81cbe206.slice/crio-073d653ea849ecc5c0916d08dee9537aa8be82b58dbecc8f7ed452bbee582013 WatchSource:0}: Error finding container 073d653ea849ecc5c0916d08dee9537aa8be82b58dbecc8f7ed452bbee582013: Status 404 returned error can't find the container with id 073d653ea849ecc5c0916d08dee9537aa8be82b58dbecc8f7ed452bbee582013
Oct 30 21:42:44.555586 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.555508    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.752440 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.752406    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:42:44.754224 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.754204    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.766601 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.766578    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.814476 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814407    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814476 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814461    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814625 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814526    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814625 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814576    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcgw\" (UniqueName: \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814749 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814705    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814750    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.814873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.814791    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.855319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855294    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:42:44.855319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855307    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:42:44.855503 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855352    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:42:44.855503 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855406    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:42:44.855503 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855440    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:42:44.855642 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.855524    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpvt6\""
Oct 30 21:42:44.856148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.856129    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:42:44.915808 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.915765    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.915943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.915833    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.915943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.915873    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.915943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.915899    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.916226 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.916023    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcgw\" (UniqueName: \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.916226 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.916078    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.916226 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.916114    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.916682 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.916659    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.916761 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.916726    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.917740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.917713    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.918214 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.918183    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.921745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.918615    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.921745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.919352    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:44.980740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:44.980715    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcgw\" (UniqueName: \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\") pod \"console-768d7c97cc-zk47r\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") " pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:45.066044 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.065988    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:45.080256 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.080233    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8zwb"
Oct 30 21:42:45.131717 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.131690    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48tph" event={"ID":"8105b15f-1a9a-4315-817f-6a6a81cbe206","Type":"ContainerStarted","Data":"dc51c190269a86177b766a52851246d5f1b3d0d8cf0858f08ee2f2d33b4a1009"}
Oct 30 21:42:45.131859 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.131738    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48tph" event={"ID":"8105b15f-1a9a-4315-817f-6a6a81cbe206","Type":"ContainerStarted","Data":"073d653ea849ecc5c0916d08dee9537aa8be82b58dbecc8f7ed452bbee582013"}
Oct 30 21:42:45.446758 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.446726    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:42:45.453280 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:45.453252    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf06763_166e_4d2b_9465_c1dee2509e3e.slice/crio-aefb5b514196eb1cdbcc3b8531c1b5004cb62c2d7e6acd74a39b7e81eaa9a4b4 WatchSource:0}: Error finding container aefb5b514196eb1cdbcc3b8531c1b5004cb62c2d7e6acd74a39b7e81eaa9a4b4: Status 404 returned error can't find the container with id aefb5b514196eb1cdbcc3b8531c1b5004cb62c2d7e6acd74a39b7e81eaa9a4b4
Oct 30 21:42:45.797715 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:45.797637    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltrb"
Oct 30 21:42:46.112927 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:46.112844    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:46.137147 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:46.137111    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48tph" event={"ID":"8105b15f-1a9a-4315-817f-6a6a81cbe206","Type":"ContainerStarted","Data":"1b7a6364ef70e043cb872dd8640486590f726a2d15696353134d32a0423af107"}
Oct 30 21:42:46.138627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:46.138601    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d7c97cc-zk47r" event={"ID":"aaf06763-166e-4d2b-9465-c1dee2509e3e","Type":"ContainerStarted","Data":"aefb5b514196eb1cdbcc3b8531c1b5004cb62c2d7e6acd74a39b7e81eaa9a4b4"}
Oct 30 21:42:46.260739 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:46.260701    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:46.753924 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:46.753869    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:48.153711 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:48.153669    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48tph" event={"ID":"8105b15f-1a9a-4315-817f-6a6a81cbe206","Type":"ContainerStarted","Data":"a02e3f6964d5548fb1d1963942732f646fb03421d1464d2c708d8cc90fa71a5c"}
Oct 30 21:42:49.158574 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:49.158497    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d7c97cc-zk47r" event={"ID":"aaf06763-166e-4d2b-9465-c1dee2509e3e","Type":"ContainerStarted","Data":"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"}
Oct 30 21:42:49.273411 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:49.273360    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-768d7c97cc-zk47r" podStartSLOduration=1.8273509190000001 podStartE2EDuration="5.273347343s" totalImagesPullingTime="3.445996424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:49.273178012 +0000 UTC m=+22.361915817" watchObservedRunningTime="2026-10-30 21:42:49.273347343 +0000 UTC m=+22.362085140"
Oct 30 21:42:49.658912 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:49.658875    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:50.525798 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:42:50.525744    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:42:50.535589 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:50.535554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:50.713849 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:50.713804    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.063875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.063835    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.215989 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.215953    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.341313 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.341240    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8bvxb"]
Oct 30 21:42:51.343183 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.343167    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366031 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366005    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-sys\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366035    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2nr\" (UniqueName: \"kubernetes.io/projected/8bd84e2c-d40a-43a5-9026-55ef6949565b-kube-api-access-mk2nr\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366052    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-textfile\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366071    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-metrics-client-ca\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366235 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366175    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-root\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366235 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366204    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-tls\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366235 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366222    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366333 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366315    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.366368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.366344    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-wtmp\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.386263 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.386233    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.395752 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.395724    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:42:51.396426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.396403    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:42:51.445089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.445025    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csjdz\""
Oct 30 21:42:51.445213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.445182    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:42:51.445265 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.445208    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:42:51.445265 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.445222    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:42:51.445265 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.445249    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:42:51.467524 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467492    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-root\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467530    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-tls\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467549    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467578    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467599    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-wtmp\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467604    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-root\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467678    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-sys\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467707    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2nr\" (UniqueName: \"kubernetes.io/projected/8bd84e2c-d40a-43a5-9026-55ef6949565b-kube-api-access-mk2nr\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467744    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-wtmp\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467751    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd84e2c-d40a-43a5-9026-55ef6949565b-sys\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467797    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-textfile\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.467956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.467826    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-metrics-client-ca\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.468248 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.468212    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-textfile\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.468352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.468332    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.468431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.468400    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bd84e2c-d40a-43a5-9026-55ef6949565b-metrics-client-ca\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.474169 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.474146    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-tls\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.474269 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.474154    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bd84e2c-d40a-43a5-9026-55ef6949565b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.527972 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.527951    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2nr\" (UniqueName: \"kubernetes.io/projected/8bd84e2c-d40a-43a5-9026-55ef6949565b-kube-api-access-mk2nr\") pod \"node-exporter-8bvxb\" (UID: \"8bd84e2c-d40a-43a5-9026-55ef6949565b\") " pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.544163 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.544119    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.656575 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.656552    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8bvxb"
Oct 30 21:42:51.666032 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:51.666006    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd84e2c_d40a_43a5_9026_55ef6949565b.slice/crio-3d91c1654afe2d94c3463f5dad3e16807c28efa7c072af785bc008884fee744f WatchSource:0}: Error finding container 3d91c1654afe2d94c3463f5dad3e16807c28efa7c072af785bc008884fee744f: Status 404 returned error can't find the container with id 3d91c1654afe2d94c3463f5dad3e16807c28efa7c072af785bc008884fee744f
Oct 30 21:42:51.697121 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.697088    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.877485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:51.877448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.169378 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:52.169339    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8bvxb" event={"ID":"8bd84e2c-d40a-43a5-9026-55ef6949565b","Type":"ContainerStarted","Data":"3d91c1654afe2d94c3463f5dad3e16807c28efa7c072af785bc008884fee744f"}
Oct 30 21:42:52.690279 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:52.690243    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.909318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:52.909282    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.124414 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.124388    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.174950 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.174912    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8bvxb" event={"ID":"8bd84e2c-d40a-43a5-9026-55ef6949565b","Type":"ContainerStarted","Data":"7b93ce29996f2092b9f52bb9438744056caf34ca78e071902c1368472f5d3e04"}
Oct 30 21:42:53.570086 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.570041    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.580969 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.580943    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:42:53.585270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.585247    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637671    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637732    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637795    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637813    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637809    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637855    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:42:53.637892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637867    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b4gn6\""
Oct 30 21:42:53.638227 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.637921    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:42:53.686420 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686389    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686551 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686425    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686551 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686442    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686551 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686465    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686589    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686632    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686668    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686825 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686720    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686804    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.686956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686918    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.687015 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.686960    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.687068 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.687048    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.687115 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.687081    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.687878 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.687856    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:42:53.737260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.737240    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:42:53.740358 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.740334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.743134 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.743116    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:42:53.787561 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787538    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787577    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787610    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787637    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787656    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787682    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787721    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787742    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787786    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.787875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787841    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788061 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787896    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788061 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787916    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788061 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.787945    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.788338    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.788521    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.788688 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.788664    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.791867 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.791842    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.792212 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.792188    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.792545 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.792531    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.792699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.792679    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.793705 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.793680    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.793942 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.793925    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.794290 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.794251    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.794362 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.794292    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.795040 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.795024    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.845724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.845680    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\") pod \"alertmanager-main-0\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:53.911770 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:53.911737    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:42:54.179862 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:54.179830    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bd84e2c-d40a-43a5-9026-55ef6949565b" containerID="7b93ce29996f2092b9f52bb9438744056caf34ca78e071902c1368472f5d3e04" exitCode=0
Oct 30 21:42:54.180013 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:54.179904    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8bvxb" event={"ID":"8bd84e2c-d40a-43a5-9026-55ef6949565b","Type":"ContainerDied","Data":"7b93ce29996f2092b9f52bb9438744056caf34ca78e071902c1368472f5d3e04"}
Oct 30 21:42:54.389925 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:54.389877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:54.396135 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:54.396106    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:42:54.404899 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:54.404870    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09d08db_d3bf_42e6_a2a9_ae9fefb24cc1.slice/crio-0f814c2b4895e735e612e92b2be032b038c792188741b3dda8f1198885b62b88 WatchSource:0}: Error finding container 0f814c2b4895e735e612e92b2be032b038c792188741b3dda8f1198885b62b88: Status 404 returned error can't find the container with id 0f814c2b4895e735e612e92b2be032b038c792188741b3dda8f1198885b62b88
Oct 30 21:42:54.556826 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:54.556717    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.066734 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.066703    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:55.067191 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.066810    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:55.072140 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.072119    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:55.185861 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.185823    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8bvxb" event={"ID":"8bd84e2c-d40a-43a5-9026-55ef6949565b","Type":"ContainerStarted","Data":"adaf12451ad1dd92d871135daff1bd2d1d13a36bf883d2c8da220d88efa614de"}
Oct 30 21:42:55.186040 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.185889    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8bvxb" event={"ID":"8bd84e2c-d40a-43a5-9026-55ef6949565b","Type":"ContainerStarted","Data":"43b50c4bf938ef9f9bf0bb9ef48301d82540e9f695ae03eb7fafa4df05c28845"}
Oct 30 21:42:55.188107 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.187237    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"0f814c2b4895e735e612e92b2be032b038c792188741b3dda8f1198885b62b88"}
Oct 30 21:42:55.191989 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.191946    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:42:55.305218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.305169    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8bvxb" podStartSLOduration=2.838339973 podStartE2EDuration="4.305153596s" totalImagesPullingTime="1.391837561s" totalInitContainerRuntime="74.976062ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:55.304222676 +0000 UTC m=+28.392960481" watchObservedRunningTime="2026-10-30 21:42:55.305153596 +0000 UTC m=+28.393891398"
Oct 30 21:42:55.415369 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.415313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.591356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.591301    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.759642 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.759565    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.916940 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.916898    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.984764 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.984736    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"]
Oct 30 21:42:55.992538 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:55.992518    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.046861 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.046747    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:42:56.047054 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.046979    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m4tn7\""
Oct 30 21:42:56.047884 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.047858    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"]
Oct 30 21:42:56.086735 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.086681    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:56.106750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.106721    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbls\" (UniqueName: \"kubernetes.io/projected/818705c7-6050-44b1-b2f7-0bae814576b5-kube-api-access-vfbls\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.106900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.106835    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/818705c7-6050-44b1-b2f7-0bae814576b5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.195946 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.195911    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5" exitCode=0
Oct 30 21:42:56.196115 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.196007    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"}
Oct 30 21:42:56.207625 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.207598    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/818705c7-6050-44b1-b2f7-0bae814576b5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.207723 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.207667    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbls\" (UniqueName: \"kubernetes.io/projected/818705c7-6050-44b1-b2f7-0bae814576b5-kube-api-access-vfbls\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.210125 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.210106    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/818705c7-6050-44b1-b2f7-0bae814576b5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.261291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.261253    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:56.265286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.265226    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbls\" (UniqueName: \"kubernetes.io/projected/818705c7-6050-44b1-b2f7-0bae814576b5-kube-api-access-vfbls\") pod \"monitoring-plugin-77f4f76594-97bpt\" (UID: \"818705c7-6050-44b1-b2f7-0bae814576b5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.306149 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.306129    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:42:56.678148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.678108    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"]
Oct 30 21:42:56.682757 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:56.682723    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818705c7_6050_44b1_b2f7_0bae814576b5.slice/crio-d2c322ba36638cfffe78f5ccbcfc06d8ccbe7efee7115a47e61e21424201b2b8 WatchSource:0}: Error finding container d2c322ba36638cfffe78f5ccbcfc06d8ccbe7efee7115a47e61e21424201b2b8: Status 404 returned error can't find the container with id d2c322ba36638cfffe78f5ccbcfc06d8ccbe7efee7115a47e61e21424201b2b8
Oct 30 21:42:56.856723 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.856691    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-755cbfd578-gd5zr"]
Oct 30 21:42:56.862124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.862098    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914019 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.913989    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-federate-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914170 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914026    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-metrics-client-ca\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914170 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914048    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914170 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-serving-certs-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914276 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914233    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914313 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914285    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914357 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914315    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8lk\" (UniqueName: \"kubernetes.io/projected/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-kube-api-access-vb8lk\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.914357 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.914352    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:56.918043 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918017    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:42:56.918160 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918085    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:42:56.918160 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918139    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-52cmv\""
Oct 30 21:42:56.918277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918185    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:42:56.918378 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918360    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:42:56.918509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.918358    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:42:56.967687 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.967611    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:42:56.969292 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:56.969255    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-755cbfd578-gd5zr"]
Oct 30 21:42:57.015113 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015073    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015131    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015172    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8lk\" (UniqueName: \"kubernetes.io/projected/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-kube-api-access-vb8lk\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015216    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015244    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-federate-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015472 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015266    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-metrics-client-ca\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015472 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015290    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.015472 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.015319    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-serving-certs-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.016061 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.016033    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-serving-certs-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.016636 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.016612    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-metrics-client-ca\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.016904 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.016879    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.018230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.018203    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.019153 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.019127    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-telemeter-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.019580 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.019554    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-secret-telemeter-client\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.020062 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.020030    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-federate-client-tls\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.072792 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.072741    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8lk\" (UniqueName: \"kubernetes.io/projected/1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e-kube-api-access-vb8lk\") pod \"telemeter-client-755cbfd578-gd5zr\" (UID: \"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e\") " pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.201092 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.201049    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt" event={"ID":"818705c7-6050-44b1-b2f7-0bae814576b5","Type":"ContainerStarted","Data":"d2c322ba36638cfffe78f5ccbcfc06d8ccbe7efee7115a47e61e21424201b2b8"}
Oct 30 21:42:57.214525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.214498    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr"
Oct 30 21:42:57.705384 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.705347    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-755cbfd578-gd5zr"]
Oct 30 21:42:57.708727 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:42:57.708697    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1423c6ea_f9b5_481c_85e4_1e3eb1f7d74e.slice/crio-742d3c164f0b491191ab6912d8ad5754c01b27381cc965557dfab1c3cfc1a102 WatchSource:0}: Error finding container 742d3c164f0b491191ab6912d8ad5754c01b27381cc965557dfab1c3cfc1a102: Status 404 returned error can't find the container with id 742d3c164f0b491191ab6912d8ad5754c01b27381cc965557dfab1c3cfc1a102
Oct 30 21:42:57.930579 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:57.930526    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:58.159822 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:58.159789    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:58.218141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:58.218102    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr" event={"ID":"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e","Type":"ContainerStarted","Data":"742d3c164f0b491191ab6912d8ad5754c01b27381cc965557dfab1c3cfc1a102"}
Oct 30 21:42:59.052596 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.052571    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pjnlr"
Oct 30 21:42:59.226008 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.225863    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"}
Oct 30 21:42:59.226008 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.225944    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"}
Oct 30 21:42:59.495846 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.495753    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:59.707261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.707100    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:59.867559 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:42:59.867475    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:00.233450 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.233412    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"}
Oct 30 21:43:00.233934 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.233485    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"}
Oct 30 21:43:00.233934 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.233527    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"}
Oct 30 21:43:00.235137 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.235105    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt" event={"ID":"818705c7-6050-44b1-b2f7-0bae814576b5","Type":"ContainerStarted","Data":"82090ebf830b5d3239c020ef53b26e63fe222142f1349afb58cad7a31d74f3e7"}
Oct 30 21:43:00.235313 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.235285    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:43:00.240459 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.240442    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt"
Oct 30 21:43:00.350260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.350202    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-97bpt" podStartSLOduration=2.823950194 podStartE2EDuration="5.350185307s" totalImagesPullingTime="2.526235113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:00.348805518 +0000 UTC m=+33.437543319" watchObservedRunningTime="2026-10-30 21:43:00.350185307 +0000 UTC m=+33.438923111"
Oct 30 21:43:00.395425 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.395395    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:00.451725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.451706    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:43:00.479198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.479173    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.522176 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.522149    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:43:00.535151 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.535130    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:43:00.535807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.535786    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:43:00.535883 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.535825    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:43:00.548584 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548561    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548691 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548594    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548691 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548626    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548844 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548688    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548844 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548762    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548849    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548919    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.548959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.548954    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549122 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549040    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549122 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549072    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549282 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549142    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549282 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549209    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549279    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549305    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549379    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549409    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmffs\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549437    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549464    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549491    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.549756 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.549541    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.553485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.553467    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:43:00.568327 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568260    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:43:00.568327 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568302    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f9tn9jvppfavt\""
Oct 30 21:43:00.568327 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568306    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:43:00.568327 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568306    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:43:00.568610 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568389    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:43:00.568610 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568427    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:43:00.568610 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.568511    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5xpkx\""
Oct 30 21:43:00.627198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.627157    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:43:00.627316 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.627272    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:43:00.627622 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.627604    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:43:00.650424 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650402    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650435    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650457    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650488    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650662 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650635    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650718 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650685    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650827 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650713    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650827 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650740    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.650827 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650802    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650834    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650887    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650915    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650939    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650968    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.650996    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651040    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651063    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651086    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651114    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmffs\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651144    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651231    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.651759 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.651734    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.652353 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.652333    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.653197 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.653160    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.653679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.653650    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.654141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.654123    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.654672 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.654628    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.654905 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.654847    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.655013 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.654921    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.655013 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.654932    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.655788 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.655376    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.655788 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.655420    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.655788 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.655723    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.656431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.656404    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.656750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.656728    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.656941 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.656921    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.677588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.677561    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:43:00.678867 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.677866    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:43:00.678867 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.677953    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:43:00.684126 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.684105    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.684435 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.684417    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.685761 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.685654    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.709113 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.709090    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmffs\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\") pod \"prometheus-k8s-0\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:00.877214 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:00.877024    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:01.240823 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.240765    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tgm2b" event={"ID":"412548c7-be96-4d51-810c-70dd2271de99","Type":"ContainerStarted","Data":"0ebf77ef2fbfd108dd0eb8e25b910a4c699310ebd8592b07142bc4dc0816f1f5"}
Oct 30 21:43:01.241829 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.241756    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:43:01.244948 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.244918    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr" event={"ID":"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e","Type":"ContainerStarted","Data":"2755a6dc88138d5adfbb3e5543416297fa2506a5522cbf9dabad1bc9a5e5cf57"}
Oct 30 21:43:01.245084 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.244962    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr" event={"ID":"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e","Type":"ContainerStarted","Data":"1433c07316d4732375542c06e8ee142a8d365261a13ecae29bd2b34ecff8b465"}
Oct 30 21:43:01.245084 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.244989    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr" event={"ID":"1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e","Type":"ContainerStarted","Data":"8587918ae0fd7b0f4d7263a0696b443e7115af9c5a1fa4bb5e4091421a17b72f"}
Oct 30 21:43:01.261431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.261404    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-tgm2b"
Oct 30 21:43:01.355838 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.355786    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-tgm2b" podStartSLOduration=1.693847045 podStartE2EDuration="21.35575261s" totalImagesPullingTime="19.661905565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:01.354481759 +0000 UTC m=+34.443219563" watchObservedRunningTime="2026-10-30 21:43:01.35575261 +0000 UTC m=+34.444490439"
Oct 30 21:43:01.368486 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.368460    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:43:01.409611 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:43:01.409568    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcca145_a3a4_4563_b653_bb95f445f93c.slice/crio-4179747fe94e809e45bd98041c780f4dbffa348e36ce47ce08f21b91ae1c748a WatchSource:0}: Error finding container 4179747fe94e809e45bd98041c780f4dbffa348e36ce47ce08f21b91ae1c748a: Status 404 returned error can't find the container with id 4179747fe94e809e45bd98041c780f4dbffa348e36ce47ce08f21b91ae1c748a
Oct 30 21:43:01.475126 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:01.475073    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-755cbfd578-gd5zr" podStartSLOduration=2.801492191 podStartE2EDuration="5.475056686s" totalImagesPullingTime="2.673564495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:01.472877555 +0000 UTC m=+34.561615360" watchObservedRunningTime="2026-10-30 21:43:01.475056686 +0000 UTC m=+34.563794489"
Oct 30 21:43:02.252994 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:02.252941    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerStarted","Data":"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"}
Oct 30 21:43:02.254991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:02.254961    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05" exitCode=0
Oct 30 21:43:02.255142 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:02.255003    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"}
Oct 30 21:43:02.255142 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:02.255083    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"4179747fe94e809e45bd98041c780f4dbffa348e36ce47ce08f21b91ae1c748a"}
Oct 30 21:43:02.379244 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:02.379179    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.422083103 podStartE2EDuration="9.37916124s" totalImagesPullingTime="5.938984388s" totalInitContainerRuntime="18.093749ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:02.378343071 +0000 UTC m=+35.467080874" watchObservedRunningTime="2026-10-30 21:43:02.37916124 +0000 UTC m=+35.467899044"
Oct 30 21:43:03.118497 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:03.118462    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66684b567d-8qpz4"
Oct 30 21:43:03.393870 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:03.393751    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.551352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:03.551309    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.709023 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:03.708944    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:07.281478 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:07.281438    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"}
Oct 30 21:43:07.282054 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:07.281517    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"}
Oct 30 21:43:07.709517 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:07.709485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:07.881909 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:07.881873    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:08.706324 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:08.706287    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:08.869833 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:08.869798    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:10.298968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.298938    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"}
Oct 30 21:43:10.299336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.299023    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"}
Oct 30 21:43:10.299336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.299047    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"}
Oct 30 21:43:10.299336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.299071    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerStarted","Data":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"}
Oct 30 21:43:10.579241 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.579180    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.234833275 podStartE2EDuration="10.579165036s" totalImagesPullingTime="7.321430343s" totalInitContainerRuntime="22.901418ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:10.52913806 +0000 UTC m=+43.617875863" watchObservedRunningTime="2026-10-30 21:43:10.579165036 +0000 UTC m=+43.667902838"
Oct 30 21:43:10.877207 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:10.877131    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:11.711977 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:11.711941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:11.879981 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:11.879940    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:12.221482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:12.221449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:12.404477 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:12.404443    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:13.476545 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:13.476500    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:13.698790 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:13.698739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:14.821835 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:14.821800    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:43:17.633752 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:17.633712    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:17.827856 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:17.827820    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:18.013311 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:18.013271    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:18.212464 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:18.212428    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:20.521334 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:43:20.521296    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:43:23.687211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:23.687178    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:23.849088 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:23.849051    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:35.451232 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:35.451197    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:35.731557 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:35.731477    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:39.841320 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:39.841259    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-768d7c97cc-zk47r" podUID="aaf06763-166e-4d2b-9465-c1dee2509e3e" containerName="console" containerID="cri-o://4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a" gracePeriod=15
Oct 30 21:43:40.130674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.130651    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:43:40.176872 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.176848    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\" (UniqueName: \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.176975 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.176878    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.176975 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.176902    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.176975 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.176943    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.176975 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.176964    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.177151 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177006    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.177151 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177053    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\") pod \"aaf06763-166e-4d2b-9465-c1dee2509e3e\" (UID: \"aaf06763-166e-4d2b-9465-c1dee2509e3e\") "
Oct 30 21:43:40.177343 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177302    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:40.177429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177335    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:40.177429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177364    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:40.177429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.177383    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:40.179106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.179076    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "kube-api-access-vdcgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:40.179215 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.179192    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:40.179258 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.179224    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config" pod "aaf06763-166e-4d2b-9465-c1dee2509e3e" (UID: "aaf06763-166e-4d2b-9465-c1dee2509e3e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:40.277744 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277718    4410 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-oauth-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277744 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277743    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdcgw\" (UniqueName: \"kubernetes.io/projected/aaf06763-166e-4d2b-9465-c1dee2509e3e-kube-api-access-vdcgw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277753    4410 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-oauth-serving-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277762    4410 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277787    4410 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf06763-166e-4d2b-9465-c1dee2509e3e-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277797    4410 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf06763-166e-4d2b-9465-c1dee2509e3e-console-serving-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.277970 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.277805    4410 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf06763-166e-4d2b-9465-c1dee2509e3e-tmp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.409494 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.409464    4410 generic.go:356] "Generic (PLEG): container finished" podID="aaf06763-166e-4d2b-9465-c1dee2509e3e" containerID="4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a" exitCode=0
Oct 30 21:43:40.409609 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.409510    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d7c97cc-zk47r"
Oct 30 21:43:40.409609 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.409551    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d7c97cc-zk47r" event={"ID":"aaf06763-166e-4d2b-9465-c1dee2509e3e","Type":"ContainerDied","Data":"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"}
Oct 30 21:43:40.409609 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.409603    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d7c97cc-zk47r" event={"ID":"aaf06763-166e-4d2b-9465-c1dee2509e3e","Type":"ContainerDied","Data":"aefb5b514196eb1cdbcc3b8531c1b5004cb62c2d7e6acd74a39b7e81eaa9a4b4"}
Oct 30 21:43:40.409701 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.409623    4410 scope.go:122] "RemoveContainer" containerID="4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"
Oct 30 21:43:40.418173 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.418152    4410 scope.go:122] "RemoveContainer" containerID="4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"
Oct 30 21:43:40.418454 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:43:40.418435    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a\": container with ID starting with 4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a not found: ID does not exist" containerID="4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"
Oct 30 21:43:40.418522 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.418461    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a"} err="failed to get container status \"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a\": rpc error: code = NotFound desc = could not find container \"4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a\": container with ID starting with 4abc628b09852367856a59d5041bf33a399f8294c70bdfb48ed73fe272541b0a not found: ID does not exist"
Oct 30 21:43:40.577069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.577043    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:43:40.582814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:40.582795    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-768d7c97cc-zk47r"]
Oct 30 21:43:41.016890 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:41.016860    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf06763-166e-4d2b-9465-c1dee2509e3e" path="/var/lib/kubelet/pods/aaf06763-166e-4d2b-9465-c1dee2509e3e/volumes"
Oct 30 21:43:48.654691 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:48.654653    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:48.696665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:48.696642    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:49.456496 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:43:49.456468    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:43:50.521112 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:43:50.521076    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:44:00.131566 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:00.131532    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:00.361542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:00.361504    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:07.106563 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.106522    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:07.301398 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.301366    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:07.497715 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.497684    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:07.676578 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.676521    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:07.867871 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.867797    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:07.868089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868068    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="alertmanager" containerID="cri-o://bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af" gracePeriod=120
Oct 30 21:44:07.868227 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868137    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-metric" containerID="cri-o://a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8" gracePeriod=120
Oct 30 21:44:07.868227 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868160    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy" containerID="cri-o://322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa" gracePeriod=120
Oct 30 21:44:07.868227 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868153    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-web" containerID="cri-o://1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac" gracePeriod=120
Oct 30 21:44:07.868468 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868249    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="prom-label-proxy" containerID="cri-o://c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd" gracePeriod=120
Oct 30 21:44:07.868468 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:07.868287    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="config-reloader" containerID="cri-o://8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade" gracePeriod=120
Oct 30 21:44:08.505699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505662    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd" exitCode=0
Oct 30 21:44:08.505699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505692    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa" exitCode=0
Oct 30 21:44:08.505699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505701    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade" exitCode=0
Oct 30 21:44:08.506152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505708    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af" exitCode=0
Oct 30 21:44:08.506152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505733    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"}
Oct 30 21:44:08.506152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505808    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"}
Oct 30 21:44:08.506152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505836    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"}
Oct 30 21:44:08.506152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:08.505860    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"}
Oct 30 21:44:09.110575 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.110551    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.195130 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195085    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195130 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195114    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195150    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195188    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195230 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195211    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195376 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195232    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195376 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195271    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195376 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195309    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195376 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195353    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195586 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195385    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195586 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195429    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195586 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195465    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.195586 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195497    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\") pod \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\" (UID: \"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1\") "
Oct 30 21:44:09.196209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195457    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:09.196209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195485    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:09.196209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.195845    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:09.198180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198152    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:09.198410 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198387    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.198481 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198463    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:09.198800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198748    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.198900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198867    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.198980 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.198961    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.199291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.199263    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "kube-api-access-kn99b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:09.199399 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.199372    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.202415 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.202395    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.208588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.208569    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config" pod "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" (UID: "f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:09.296959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.296936    4410 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-out\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.296959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.296959    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-kube-api-access-kn99b\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.296969    4410 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.296980    4410 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.296990    4410 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297000    4410 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-main-tls\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297009    4410 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297018    4410 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297026    4410 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-metrics-client-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297034    4410 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-tls-assets\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297043    4410 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-web-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297052    4410 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-alertmanager-main-db\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.297106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.297060    4410 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1-cluster-tls-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:09.511962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.511879    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8" exitCode=0
Oct 30 21:44:09.511962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.511911    4410 generic.go:356] "Generic (PLEG): container finished" podID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerID="1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac" exitCode=0
Oct 30 21:44:09.511962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.511944    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"}
Oct 30 21:44:09.511962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.511954    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.512488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.512016    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"}
Oct 30 21:44:09.512488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.512062    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1","Type":"ContainerDied","Data":"0f814c2b4895e735e612e92b2be032b038c792188741b3dda8f1198885b62b88"}
Oct 30 21:44:09.512488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.512090    4410 scope.go:122] "RemoveContainer" containerID="c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"
Oct 30 21:44:09.519865 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.519846    4410 scope.go:122] "RemoveContainer" containerID="a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"
Oct 30 21:44:09.526487 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.526472    4410 scope.go:122] "RemoveContainer" containerID="322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"
Oct 30 21:44:09.532972 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.532955    4410 scope.go:122] "RemoveContainer" containerID="1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"
Oct 30 21:44:09.539030 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.539014    4410 scope.go:122] "RemoveContainer" containerID="8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"
Oct 30 21:44:09.545309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.545292    4410 scope.go:122] "RemoveContainer" containerID="bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"
Oct 30 21:44:09.551705 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.551689    4410 scope.go:122] "RemoveContainer" containerID="dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"
Oct 30 21:44:09.562184 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.562167    4410 scope.go:122] "RemoveContainer" containerID="c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"
Oct 30 21:44:09.562450 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.562428    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd\": container with ID starting with c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd not found: ID does not exist" containerID="c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"
Oct 30 21:44:09.562534 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.562459    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"} err="failed to get container status \"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd\": rpc error: code = NotFound desc = could not find container \"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd\": container with ID starting with c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd not found: ID does not exist"
Oct 30 21:44:09.562534 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.562479    4410 scope.go:122] "RemoveContainer" containerID="a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"
Oct 30 21:44:09.562711 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.562690    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8\": container with ID starting with a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8 not found: ID does not exist" containerID="a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"
Oct 30 21:44:09.562799 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.562722    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"} err="failed to get container status \"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8\": rpc error: code = NotFound desc = could not find container \"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8\": container with ID starting with a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8 not found: ID does not exist"
Oct 30 21:44:09.562799 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.562744    4410 scope.go:122] "RemoveContainer" containerID="322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"
Oct 30 21:44:09.562998 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.562981    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa\": container with ID starting with 322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa not found: ID does not exist" containerID="322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"
Oct 30 21:44:09.563033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563003    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"} err="failed to get container status \"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa\": rpc error: code = NotFound desc = could not find container \"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa\": container with ID starting with 322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa not found: ID does not exist"
Oct 30 21:44:09.563033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563020    4410 scope.go:122] "RemoveContainer" containerID="1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"
Oct 30 21:44:09.563264 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.563245    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac\": container with ID starting with 1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac not found: ID does not exist" containerID="1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"
Oct 30 21:44:09.563328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563271    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"} err="failed to get container status \"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac\": rpc error: code = NotFound desc = could not find container \"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac\": container with ID starting with 1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac not found: ID does not exist"
Oct 30 21:44:09.563328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563290    4410 scope.go:122] "RemoveContainer" containerID="8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"
Oct 30 21:44:09.563531 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.563513    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade\": container with ID starting with 8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade not found: ID does not exist" containerID="8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"
Oct 30 21:44:09.563567 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563544    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"} err="failed to get container status \"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade\": rpc error: code = NotFound desc = could not find container \"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade\": container with ID starting with 8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade not found: ID does not exist"
Oct 30 21:44:09.563567 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563558    4410 scope.go:122] "RemoveContainer" containerID="bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"
Oct 30 21:44:09.563741 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.563724    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af\": container with ID starting with bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af not found: ID does not exist" containerID="bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"
Oct 30 21:44:09.564172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563747    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"} err="failed to get container status \"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af\": rpc error: code = NotFound desc = could not find container \"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af\": container with ID starting with bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af not found: ID does not exist"
Oct 30 21:44:09.564172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.563765    4410 scope.go:122] "RemoveContainer" containerID="dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"
Oct 30 21:44:09.564172 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:09.564035    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5\": container with ID starting with dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5 not found: ID does not exist" containerID="dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"
Oct 30 21:44:09.564172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564060    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"} err="failed to get container status \"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5\": rpc error: code = NotFound desc = could not find container \"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5\": container with ID starting with dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5 not found: ID does not exist"
Oct 30 21:44:09.564172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564079    4410 scope.go:122] "RemoveContainer" containerID="c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"
Oct 30 21:44:09.564442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564316    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd"} err="failed to get container status \"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd\": rpc error: code = NotFound desc = could not find container \"c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd\": container with ID starting with c71962367f36996e1ed3708f4afccaa2b442adc838cc60178fcf2ca4ef8fcadd not found: ID does not exist"
Oct 30 21:44:09.564442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564336    4410 scope.go:122] "RemoveContainer" containerID="a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"
Oct 30 21:44:09.564573 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564553    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8"} err="failed to get container status \"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8\": rpc error: code = NotFound desc = could not find container \"a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8\": container with ID starting with a07ea4983bcdc76a40288b68e3b226c6a3648fb7a65ae2f5f1e3c3a0389061e8 not found: ID does not exist"
Oct 30 21:44:09.564573 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564571    4410 scope.go:122] "RemoveContainer" containerID="322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"
Oct 30 21:44:09.564802 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564757    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa"} err="failed to get container status \"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa\": rpc error: code = NotFound desc = could not find container \"322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa\": container with ID starting with 322ed4eaf3795c165e4db9ec8784c3435fcfc443bdbdcff35b49f79bf234feaa not found: ID does not exist"
Oct 30 21:44:09.564802 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564800    4410 scope.go:122] "RemoveContainer" containerID="1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"
Oct 30 21:44:09.564995 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564976    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac"} err="failed to get container status \"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac\": rpc error: code = NotFound desc = could not find container \"1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac\": container with ID starting with 1d1d1a6c98925369d49593b99d759f224fc1926ffcabaa30092ac16a7a98d3ac not found: ID does not exist"
Oct 30 21:44:09.565057 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.564996    4410 scope.go:122] "RemoveContainer" containerID="8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"
Oct 30 21:44:09.565192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.565169    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade"} err="failed to get container status \"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade\": rpc error: code = NotFound desc = could not find container \"8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade\": container with ID starting with 8f2d5bab3ce9e6adfdd6f1e56cc9e400ccf0e88875b0be9056cf1fd3462f1ade not found: ID does not exist"
Oct 30 21:44:09.565253 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.565192    4410 scope.go:122] "RemoveContainer" containerID="bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"
Oct 30 21:44:09.565371 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.565355    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af"} err="failed to get container status \"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af\": rpc error: code = NotFound desc = could not find container \"bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af\": container with ID starting with bc0a6a6644b0fd6a0c23273a80b8015f5b463dc06004cf594011fe31b3d2b8af not found: ID does not exist"
Oct 30 21:44:09.565431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.565371    4410 scope.go:122] "RemoveContainer" containerID="dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"
Oct 30 21:44:09.565547 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.565532    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5"} err="failed to get container status \"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5\": rpc error: code = NotFound desc = could not find container \"dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5\": container with ID starting with dc02744e84ec39e799a6f4dd96f6427445be078b5523b01fbbab64940cf523e5 not found: ID does not exist"
Oct 30 21:44:09.681099 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.681062    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:09.687704 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.687682    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:09.730724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.730697    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:09.730993 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.730980    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="config-reloader"
Oct 30 21:44:09.730993 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.730993    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="config-reloader"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731006    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-metric"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731011    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-metric"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731020    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731026    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731033    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="prom-label-proxy"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731037    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="prom-label-proxy"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731045    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="alertmanager"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731051    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="alertmanager"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731067    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="init-config-reloader"
Oct 30 21:44:09.731071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731072    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="init-config-reloader"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731081    4410 cpu_manager.go:414] "containerMap: removing container" podUID="aaf06763-166e-4d2b-9465-c1dee2509e3e" containerName="console"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731086    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf06763-166e-4d2b-9465-c1dee2509e3e" containerName="console"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731092    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-web"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731096    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-web"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731142    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf06763-166e-4d2b-9465-c1dee2509e3e" containerName="console"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731150    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-web"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731155    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="config-reloader"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731161    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy-metric"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731168    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="kube-rbac-proxy"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731172    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="prom-label-proxy"
Oct 30 21:44:09.731335 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.731177    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" containerName="alertmanager"
Oct 30 21:44:09.734426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.734410    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.787328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787265    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:44:09.787610 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787592    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:44:09.787699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787640    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:44:09.787753 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787693    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:44:09.787815 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787793    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:44:09.787880 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787813    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:44:09.787880 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787850    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:44:09.787982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787929    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:44:09.787982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.787967    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b4gn6\""
Oct 30 21:44:09.801100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801081    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801182 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801112    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-config-out\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801182 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801140    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801237    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801285 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801265    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-config-volume\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801325 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801284    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801379 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801325    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801379 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801357    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801385    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdjf\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-kube-api-access-9jdjf\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801535 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801479    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-web-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801575 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801530    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801585    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.801639 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.801614    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.837145 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.837125    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:44:09.855574 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.855549    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:09.902350 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902321    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-config-volume\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902350 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902353    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902371    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902506    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902594 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902545    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdjf\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-kube-api-access-9jdjf\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902630 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902602    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-web-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902695 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902672    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902717    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902742    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902876 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902840    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.902876 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902865    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-config-out\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.903030 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902889    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.903030 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.902918    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.903310 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.903282    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.903845 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.903798    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.904576 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.904547    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a8289d-32a6-4839-a865-643fea03e494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.905506 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.905479    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.905598 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.905484    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-config-volume\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.905900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.905877    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.905991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.905971    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.906186 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.906168    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-web-config\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.906357 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.906337    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.906394 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.906338    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.906792 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.906754    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25a8289d-32a6-4839-a865-643fea03e494-config-out\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.907192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.907176    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25a8289d-32a6-4839-a865-643fea03e494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:09.960058 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:09.960031    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdjf\" (UniqueName: \"kubernetes.io/projected/25a8289d-32a6-4839-a865-643fea03e494-kube-api-access-9jdjf\") pod \"alertmanager-main-0\" (UID: \"25a8289d-32a6-4839-a865-643fea03e494\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:10.054209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:10.053742    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:44:10.498740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:10.498555    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:44:10.501306 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:44:10.501275    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a8289d_32a6_4839_a865_643fea03e494.slice/crio-02014c01adcdbc28417992c8bca71b4565f6fdd1447f64bbcb0426f14bb9d142 WatchSource:0}: Error finding container 02014c01adcdbc28417992c8bca71b4565f6fdd1447f64bbcb0426f14bb9d142: Status 404 returned error can't find the container with id 02014c01adcdbc28417992c8bca71b4565f6fdd1447f64bbcb0426f14bb9d142
Oct 30 21:44:10.516209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:10.516179    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"02014c01adcdbc28417992c8bca71b4565f6fdd1447f64bbcb0426f14bb9d142"}
Oct 30 21:44:11.016821 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:11.016793    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1" path="/var/lib/kubelet/pods/f09d08db-d3bf-42e6-a2a9-ae9fefb24cc1/volumes"
Oct 30 21:44:11.521398 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:11.521363    4410 generic.go:356] "Generic (PLEG): container finished" podID="25a8289d-32a6-4839-a865-643fea03e494" containerID="b2152377ed6192e4d84fd46806e7b99f795b187b57964ec634cf652c340e09fa" exitCode=0
Oct 30 21:44:11.521760 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:11.521451    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerDied","Data":"b2152377ed6192e4d84fd46806e7b99f795b187b57964ec634cf652c340e09fa"}
Oct 30 21:44:12.528297 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528258    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"b03dca31d5863a6011d1d18e8172e94e0776b540e7fcf2018d951729146adac8"}
Oct 30 21:44:12.528664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528372    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"7c8a3ce3885c9080ebb2d79a26988f6f81fc7e65bd851144cac60d6bc1419873"}
Oct 30 21:44:12.528664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528414    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"af430564cf442bc3beee82883db9edc581e1ee4097a094f7ddb79075495554b4"}
Oct 30 21:44:12.528664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528453    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"5a199429019896b6a8b348630406e945a1dd11c60d76d18b6de7e8ab33698041"}
Oct 30 21:44:12.528664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528493    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"8625b0b62d693d3c2cc7664df487be2b1e7e49c49c98d2e6daca1b7e3d3c351f"}
Oct 30 21:44:12.528664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.528556    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25a8289d-32a6-4839-a865-643fea03e494","Type":"ContainerStarted","Data":"59e76f0ad67b859362259b24bab18c4183ae8d09bbdf1ba92dbe053c155fdd41"}
Oct 30 21:44:12.656926 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.656872    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.644481732 podStartE2EDuration="3.656859448s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.377716ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:12.654976237 +0000 UTC m=+105.743714040" watchObservedRunningTime="2026-10-30 21:44:12.656859448 +0000 UTC m=+105.745597251"
Oct 30 21:44:12.730337 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.730308    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:12.884708 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:12.884632    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:13.638793 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.638745    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:13.640152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640090    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="prometheus" containerID="cri-o://87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e" gracePeriod=600
Oct 30 21:44:13.640463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640436    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20" gracePeriod=600
Oct 30 21:44:13.640569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640551    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy" containerID="cri-o://eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3" gracePeriod=600
Oct 30 21:44:13.640646 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640630    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-web" containerID="cri-o://d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73" gracePeriod=600
Oct 30 21:44:13.640717 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640702    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="thanos-sidecar" containerID="cri-o://1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55" gracePeriod=600
Oct 30 21:44:13.640797 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.640767    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="config-reloader" containerID="cri-o://0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a" gracePeriod=600
Oct 30 21:44:13.816524 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.816493    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:13.882442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:13.882420    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.038489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038411    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038456    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038501    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038525    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038554    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038585    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038609    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038658    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.038696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038686    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038751    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038800    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038827    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038855    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038882    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038906    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038940    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038970    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.038997    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039032    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039065    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039096    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\") pod \"8bcca145-a3a4-4563-b653-bb95f445f93c\" (UID: \"8bcca145-a3a4-4563-b653-bb95f445f93c\") "
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039200    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039211    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039367    4410 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039441    4410 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039456    4410 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-metrics-client-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039545    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.039585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.039561    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.040801 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.040732    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.041594 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.041568    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:14.042069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.042044    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:14.042458 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.042431    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.042700 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.042666    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.042901 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.042741    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.043146 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.043113    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.043321 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.043300    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:14.043600 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.043568    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:14.043840 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.043816    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.044052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.044028    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "kube-api-access-fmffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:14.044151 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.044029    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.044242 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.044214    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.044336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.044319    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.053279 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.053255    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config" pod "8bcca145-a3a4-4563-b653-bb95f445f93c" (UID: "8bcca145-a3a4-4563-b653-bb95f445f93c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:14.139892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139871    4410 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-web-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.139892 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139891    4410 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139904    4410 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-kube-rbac-proxy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139913    4410 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139923    4410 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139933    4410 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-metrics-client-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139942    4410 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139951    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmffs\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-kube-api-access-fmffs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139959    4410 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139968    4410 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-config\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139976    4410 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139986    4410 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-grpc-tls\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.139994    4410 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-config-out\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.140002    4410 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcca145-a3a4-4563-b653-bb95f445f93c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.140012    4410 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bcca145-a3a4-4563-b653-bb95f445f93c-tls-assets\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.140021    4410 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bcca145-a3a4-4563-b653-bb95f445f93c-prometheus-k8s-db\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.140339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.140029    4410 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bcca145-a3a4-4563-b653-bb95f445f93c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:14.538873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538839    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20" exitCode=0
Oct 30 21:44:14.538873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538867    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3" exitCode=0
Oct 30 21:44:14.538873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538876    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73" exitCode=0
Oct 30 21:44:14.538873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538881    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55" exitCode=0
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538888    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a" exitCode=0
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538893    4410 generic.go:356] "Generic (PLEG): container finished" podID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e" exitCode=0
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538914    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.538925    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539013    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539053    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539076    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539094    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539135    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539177    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"}
Oct 30 21:44:14.539211 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.539216    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bcca145-a3a4-4563-b653-bb95f445f93c","Type":"ContainerDied","Data":"4179747fe94e809e45bd98041c780f4dbffa348e36ce47ce08f21b91ae1c748a"}
Oct 30 21:44:14.546623 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.546550    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.553162 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.553141    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.559435 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.559419    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.565726 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.565710    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.571841 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.571826    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.578719 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.578694    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.586530 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.586510    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.586828 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.586810    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.586881 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.586838    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.586881 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.586856    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.587133 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.587112    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.587179 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587139    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.587179 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587155    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.587397 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.587381    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.587447 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587401    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.587447 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587413    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.587637 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.587620    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.587681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587642    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.587681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587656    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.587889 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.587870    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.587931 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587892    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.587931 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.587904    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.588120 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.588106    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.588171 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588123    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.588171 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588134    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.588345 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:14.588330    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.588389 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588347    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.588389 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588358    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.588566 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588544    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.588616 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588566    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.588832 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588750    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.588832 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588766    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.588977 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588965    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.589027 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.588979    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.589174 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589156    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.589213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589175    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.589391 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589374    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.589391 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589390    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.589591 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589574    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.589591 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589590    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.589800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589768    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.589800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589797    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.589976 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589960    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.590016 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.589975    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.590210 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590192    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.590254 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590211    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.590427 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590408    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.590464 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590429    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.590619 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590601    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.590656 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590619    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.590848 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590828    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.590901 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.590849    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.591050 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591031    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.591089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591051    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.591301 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591283    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.591339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591302    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.591482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591466    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.591517 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591482    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.591653 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591636    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.591690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591654    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.591890 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591870    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.591929 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.591890    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.592097 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592081    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.592133 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592097    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.592308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592287    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.592357 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592308    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.592523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592507    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.592523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592522    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.592732 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592716    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.592769 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592733    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.592940 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592922    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.592981 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.592939    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.593148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593133    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.593189 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593147    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.593337 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593321    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.593377 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593337    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.593524 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593508    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.593564 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593523    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.593724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593707    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.593724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593723    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.593947 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593924    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.593947 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.593946    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.594159 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594140    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.594209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594160    4410 scope.go:122] "RemoveContainer" containerID="48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"
Oct 30 21:44:14.594360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594344    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20"} err="failed to get container status \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": rpc error: code = NotFound desc = could not find container \"48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20\": container with ID starting with 48f8a0fba3a78369afa85377a8a3bdb5dcaaebb4fbbbd43547cbffd0b388ab20 not found: ID does not exist"
Oct 30 21:44:14.594360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594359    4410 scope.go:122] "RemoveContainer" containerID="eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"
Oct 30 21:44:14.594552 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594535    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3"} err="failed to get container status \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": rpc error: code = NotFound desc = could not find container \"eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3\": container with ID starting with eff2bc5fd8374cef95fc70fa73a72429926c3e6831a924d047620611ac84fbb3 not found: ID does not exist"
Oct 30 21:44:14.594599 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594551    4410 scope.go:122] "RemoveContainer" containerID="d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"
Oct 30 21:44:14.594762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594746    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73"} err="failed to get container status \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": rpc error: code = NotFound desc = could not find container \"d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73\": container with ID starting with d8d1ddb49b8d70959b3b6f008b9c422b1d69e38e10ac14fdbf100ca34689ec73 not found: ID does not exist"
Oct 30 21:44:14.594836 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594761    4410 scope.go:122] "RemoveContainer" containerID="1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"
Oct 30 21:44:14.594985 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594970    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55"} err="failed to get container status \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": rpc error: code = NotFound desc = could not find container \"1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55\": container with ID starting with 1d3f40ddd8a52c56f3c0b45f2b08a7da1887b089325e93dab45d8789efe98b55 not found: ID does not exist"
Oct 30 21:44:14.595029 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.594985    4410 scope.go:122] "RemoveContainer" containerID="0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"
Oct 30 21:44:14.595172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.595157    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a"} err="failed to get container status \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": rpc error: code = NotFound desc = could not find container \"0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a\": container with ID starting with 0aa6afe49ab090ac73e55b7413d075cc22c3d761be7407e865fe3fbb5e83729a not found: ID does not exist"
Oct 30 21:44:14.595215 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.595172    4410 scope.go:122] "RemoveContainer" containerID="87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"
Oct 30 21:44:14.595350 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.595336    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e"} err="failed to get container status \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": rpc error: code = NotFound desc = could not find container \"87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e\": container with ID starting with 87250e0bedbfd8c941b8ec1815e3ce44301cddbf74129c493000b02475625c4e not found: ID does not exist"
Oct 30 21:44:14.595397 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.595350    4410 scope.go:122] "RemoveContainer" containerID="90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"
Oct 30 21:44:14.595542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.595527    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05"} err="failed to get container status \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": rpc error: code = NotFound desc = could not find container \"90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05\": container with ID starting with 90b993de0aecff5b02f7fd49d957ac47d66768dabf65161bad033d01dca82a05 not found: ID does not exist"
Oct 30 21:44:14.755794 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.755761    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:14.805281 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.805222    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:14.806080 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806060    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:14.806325 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806314    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="init-config-reloader"
Oct 30 21:44:14.806360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806327    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="init-config-reloader"
Oct 30 21:44:14.806360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806342    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="prometheus"
Oct 30 21:44:14.806360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806347    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="prometheus"
Oct 30 21:44:14.806360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806354    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-web"
Oct 30 21:44:14.806360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806359    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-web"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806365    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="config-reloader"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806370    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="config-reloader"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806379    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806384    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806399    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-thanos"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806404    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-thanos"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806413    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="thanos-sidecar"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806417    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="thanos-sidecar"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806456    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806465    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-thanos"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806471    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="thanos-sidecar"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806477    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="prometheus"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806483    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="kube-rbac-proxy-web"
Oct 30 21:44:14.806499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.806488    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" containerName="config-reloader"
Oct 30 21:44:14.809434 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.809416    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.862360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862341    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:44:14.862461 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862409    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:44:14.862511 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862492    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:44:14.862790 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862763    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:44:14.862868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862855    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:44:14.862919 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862909    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:44:14.862962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862935    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:44:14.863034 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.862995    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:44:14.863285 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.863268    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f9tn9jvppfavt\""
Oct 30 21:44:14.863350 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.863324    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:44:14.863416 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.863345    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:44:14.863416 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.863358    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:44:14.910596 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.910578    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5xpkx\""
Oct 30 21:44:14.910647 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.910582    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:44:14.913041 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.913017    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:44:14.913118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.913067    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:44:14.913118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.913097    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:44:14.927879 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.927858    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:14.945665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945643    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945674    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945698    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945858 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945812    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945858 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945846    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945967 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945922    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.945967 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945955    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.945974    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946000    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946026    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946122 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946093    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946122    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmds\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-kube-api-access-bdmds\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946158    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946315 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946241    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946315 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946273    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946315 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946298    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946390    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946418    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:14.946463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:14.946436    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.017124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.017097    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcca145-a3a4-4563-b653-bb95f445f93c" path="/var/lib/kubelet/pods/8bcca145-a3a4-4563-b653-bb95f445f93c/volumes"
Oct 30 21:44:15.047569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047547    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047582    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047600    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmds\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-kube-api-access-bdmds\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047622    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047648    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047665    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047686    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047715    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047984 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047738    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047984 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047760    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047984 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047821    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.047984 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047852    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048187 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048079    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048187 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.047876    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048289 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048217    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048289 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048246    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048289 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048278    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048432 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048302    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048432 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048336    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048432 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048371    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.048582 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048560    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.049234 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.048931    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.049444 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.049418    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.049523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.049449    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.049588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.049526    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.051328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.049977    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.051328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.050681    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.051328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.050796    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.051328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.051281    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.051328 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.051298    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.052201 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.051839    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.052201 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.051945    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.052965 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.052921    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.053504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.053082    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.053504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.053255    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.053504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.053384    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cab40cc-a828-481e-b0a6-a5260b0a1cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.053504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.053465    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.053712 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.053623    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cab40cc-a828-481e-b0a6-a5260b0a1cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.054542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.054524    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cab40cc-a828-481e-b0a6-a5260b0a1cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.104921 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.104879    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmds\" (UniqueName: \"kubernetes.io/projected/2cab40cc-a828-481e-b0a6-a5260b0a1cde-kube-api-access-bdmds\") pod \"prometheus-k8s-0\" (UID: \"2cab40cc-a828-481e-b0a6-a5260b0a1cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.124761 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.124738    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:15.555301 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.555263    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:44:15.558857 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:44:15.558835    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cab40cc_a828_481e_b0a6_a5260b0a1cde.slice/crio-51d01957a518c5e61f9b99a7119acc9c846828ab9cf3d1f7fee607dcdf12530c WatchSource:0}: Error finding container 51d01957a518c5e61f9b99a7119acc9c846828ab9cf3d1f7fee607dcdf12530c: Status 404 returned error can't find the container with id 51d01957a518c5e61f9b99a7119acc9c846828ab9cf3d1f7fee607dcdf12530c
Oct 30 21:44:15.689345 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.689295    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:15.902475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:15.902440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:16.548133 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:16.548101    4410 generic.go:356] "Generic (PLEG): container finished" podID="2cab40cc-a828-481e-b0a6-a5260b0a1cde" containerID="942972396aebd32b252ae136589733fa7de183d69a1377ff794e87e68e439686" exitCode=0
Oct 30 21:44:16.548309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:16.548146    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerDied","Data":"942972396aebd32b252ae136589733fa7de183d69a1377ff794e87e68e439686"}
Oct 30 21:44:16.548309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:16.548198    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"51d01957a518c5e61f9b99a7119acc9c846828ab9cf3d1f7fee607dcdf12530c"}
Oct 30 21:44:17.555001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.554963    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"6bd52bf7920c81d2320c903a1627340da8161f97dd08fad03d4e88701d1daf39"}
Oct 30 21:44:17.555475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.555039    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"ed12d9493dfc4d8c95d80defc8a0b7e6ff11e3bcaa4d56d534550c7ff67222f1"}
Oct 30 21:44:17.555475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.555078    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"f2ba2e38a5123f583b569e6c9015be6966c30a4d879b75b31fe81550b9f63f7b"}
Oct 30 21:44:17.555475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.555118    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"96f13c9feebbdd8f3f5264587ea92131e69153d584a5785f9989001cd8144266"}
Oct 30 21:44:17.555475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.555160    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"309d3b1a93708f589078b8edbce675835ba873718e197f52f15dfdd936063910"}
Oct 30 21:44:17.555475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.555199    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cab40cc-a828-481e-b0a6-a5260b0a1cde","Type":"ContainerStarted","Data":"2c533e821a4228df84b765cb99269fec8dbc84a03d0e85ab24bcf357ecdbe102"}
Oct 30 21:44:17.683814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:17.683742    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.666597145 podStartE2EDuration="3.68372893s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.131785ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:17.682896974 +0000 UTC m=+110.771634779" watchObservedRunningTime="2026-10-30 21:44:17.68372893 +0000 UTC m=+110.772466733"
Oct 30 21:44:20.125129 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:20.125094    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:44:20.521531 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:20.521490    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:44:25.805128 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:25.805089    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:26.433602 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:26.433568    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:37.090368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:37.090331    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:37.774418 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:44:37.774379    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:50.520567 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:44:50.520484    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:45:14.133478 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:14.133437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:15.125933 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:15.125898    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:45:15.141347 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:15.141323    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:45:15.764364 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:15.764335    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:45:20.521133 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:45:20.521097    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:45:35.929745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:35.929700    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:50.520960 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:45:50.520922    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:45:58.580650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:58.580607    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:58.759243 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:58.759208    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:58.919504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:45:58.919465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.195670 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:00.195628    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.410674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:00.410632    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:01.314217 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:01.314188    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:06.008518 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:06.008483    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:06.162858 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:06.162827    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:06.486419 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:06.486384    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:06.680403 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:06.680366    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:06.826828 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:06.826794    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:10.639902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:10.639865    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:14.497618 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:14.497579    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:20.521640 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:46:20.521602    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:46:33.948536 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:33.948494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:34.106205 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:34.106176    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:36.896425 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:36.896394    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:42.132974 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:42.132939    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.187874 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:43.187837    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:44.159467 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:44.159438    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:48.122582 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:48.122549    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:49.947301 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:49.947261    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:50.522795 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:46:50.522746    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:46:53.070734 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:53.070697    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.333676 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:53.333589    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.555082 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:53.555046    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.724151 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:53.724118    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.118587 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:54.118509    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.332929 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:54.332894    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.926488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:54.926449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.407520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.407489    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"]
Oct 30 21:46:56.410562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.410547    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.454431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.454402    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.466504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.466483    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:46:56.466560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.466491    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:46:56.514306 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.514286    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-xgd4h\""
Oct 30 21:46:56.515323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.515305    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"]
Oct 30 21:46:56.521560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.521541    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50428e7-a958-4924-94d7-c4d2c7a1cff6-tmp\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.521618 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.521604    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfn4g\" (UniqueName: \"kubernetes.io/projected/c50428e7-a958-4924-94d7-c4d2c7a1cff6-kube-api-access-xfn4g\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.622884 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.622857    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfn4g\" (UniqueName: \"kubernetes.io/projected/c50428e7-a958-4924-94d7-c4d2c7a1cff6-kube-api-access-xfn4g\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.622992 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.622901    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50428e7-a958-4924-94d7-c4d2c7a1cff6-tmp\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.623285 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.623268    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50428e7-a958-4924-94d7-c4d2c7a1cff6-tmp\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.680587 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.680533    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfn4g\" (UniqueName: \"kubernetes.io/projected/c50428e7-a958-4924-94d7-c4d2c7a1cff6-kube-api-access-xfn4g\") pod \"jobset-operator-674cd59dc5-mx7rw\" (UID: \"c50428e7-a958-4924-94d7-c4d2c7a1cff6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:56.712740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.712714    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.735690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:56.735658    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"
Oct 30 21:46:57.110446 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:57.110376    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw"]
Oct 30 21:46:57.113791 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:46:57.113746    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc50428e7_a958_4924_94d7_c4d2c7a1cff6.slice/crio-b59780290af11d9dc9a32132ee86b8a91c3ce9f222c4c68e5aa6adf6ccaf1d5a WatchSource:0}: Error finding container b59780290af11d9dc9a32132ee86b8a91c3ce9f222c4c68e5aa6adf6ccaf1d5a: Status 404 returned error can't find the container with id b59780290af11d9dc9a32132ee86b8a91c3ce9f222c4c68e5aa6adf6ccaf1d5a
Oct 30 21:46:58.090721 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:58.090690    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw" event={"ID":"c50428e7-a958-4924-94d7-c4d2c7a1cff6","Type":"ContainerStarted","Data":"b59780290af11d9dc9a32132ee86b8a91c3ce9f222c4c68e5aa6adf6ccaf1d5a"}
Oct 30 21:46:58.466756 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:58.466713    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:58.704382 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:58.704344    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:58.881375 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:58.881281    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:59.151853 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.151699    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:59.319732 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.319698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:59.381332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.381300    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"]
Oct 30 21:46:59.384696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.384677    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.437080 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.437027    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:46:59.437197 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.437089    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:46:59.444529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.444497    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltln\" (UniqueName: \"kubernetes.io/projected/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-kube-api-access-pltln\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.444747 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.444648    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.444747 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.444709    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.469472 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.469440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:59.486326 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.486306    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:46:59.487948 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.487896    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"]
Oct 30 21:46:59.535850 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.535802    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:46:59.535850 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.535813    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqkzq\""
Oct 30 21:46:59.545336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.545313    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.545460 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.545355    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.545460 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.545379    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltln\" (UniqueName: \"kubernetes.io/projected/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-kube-api-access-pltln\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.545460 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:46:59.545453    4410 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Oct 30 21:46:59.545632 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:46:59.545521    4410 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert podName:746521d2-0f3e-4cd3-bc9f-0389bf2ea65d nodeName:}" failed. No retries permitted until 2026-10-30 21:47:00.045500385 +0000 UTC m=+273.134238196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-t4lgz" (UID: "746521d2-0f3e-4cd3-bc9f-0389bf2ea65d") : secret "plugin-serving-cert" not found
Oct 30 21:46:59.546011 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.545992    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.609125 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.609097    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltln\" (UniqueName: \"kubernetes.io/projected/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-kube-api-access-pltln\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:46:59.725433 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:46:59.725363    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:00.050322 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:00.050242    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:47:00.052956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:00.052928    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/746521d2-0f3e-4cd3-bc9f-0389bf2ea65d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4lgz\" (UID: \"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:47:00.297463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:00.297425    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"
Oct 30 21:47:00.677333 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:00.677265    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz"]
Oct 30 21:47:00.681516 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:00.681487    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod746521d2_0f3e_4cd3_bc9f_0389bf2ea65d.slice/crio-dc2491d9cc886339eb210a046b0c9c2a591ff232f0d201f7a76304e535e59a0d WatchSource:0}: Error finding container dc2491d9cc886339eb210a046b0c9c2a591ff232f0d201f7a76304e535e59a0d: Status 404 returned error can't find the container with id dc2491d9cc886339eb210a046b0c9c2a591ff232f0d201f7a76304e535e59a0d
Oct 30 21:47:00.812492 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:00.812461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:01.103697 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.103611    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw" event={"ID":"c50428e7-a958-4924-94d7-c4d2c7a1cff6","Type":"ContainerStarted","Data":"5ccb61dea4457016c3f1af214e12fb9743ad69d380cfe3b0845a2b656d80ce13"}
Oct 30 21:47:01.104796 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.104757    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz" event={"ID":"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d","Type":"ContainerStarted","Data":"dc2491d9cc886339eb210a046b0c9c2a591ff232f0d201f7a76304e535e59a0d"}
Oct 30 21:47:01.118388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.118361    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"]
Oct 30 21:47:01.121627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.121613    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.173876 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.173853    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:47:01.173967 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.173948    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:47:01.204722 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.204697    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v8d8v\""
Oct 30 21:47:01.206027 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.206010    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"]
Oct 30 21:47:01.223883 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.223840    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mx7rw" podStartSLOduration=1.468799975 podStartE2EDuration="5.223827397s" totalImagesPullingTime="3.755027422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:01.216487859 +0000 UTC m=+274.305225662" watchObservedRunningTime="2026-10-30 21:47:01.223827397 +0000 UTC m=+274.312565197"
Oct 30 21:47:01.261815 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.261766    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5xq\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-kube-api-access-nh5xq\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.261907 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.261874    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.271258 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.271233    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:01.363014 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.362948    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5xq\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-kube-api-access-nh5xq\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.363392 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.363019    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.421676 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.421647    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.422405 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.422383    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5xq\" (UniqueName: \"kubernetes.io/projected/89b4de6d-c0d7-4a35-a83f-1ce3a61b6636-kube-api-access-nh5xq\") pod \"cert-manager-cainjector-5c57bbd4fc-jrmz4\" (UID: \"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.432176 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.432159    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"
Oct 30 21:47:01.590221 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.590188    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:01.816318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.816282    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4"]
Oct 30 21:47:01.819956 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:01.819925    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b4de6d_c0d7_4a35_a83f_1ce3a61b6636.slice/crio-26d1a16f13073cb57aa8a693b5bd87c1bedc6e607056e71783c577c8b7660531 WatchSource:0}: Error finding container 26d1a16f13073cb57aa8a693b5bd87c1bedc6e607056e71783c577c8b7660531: Status 404 returned error can't find the container with id 26d1a16f13073cb57aa8a693b5bd87c1bedc6e607056e71783c577c8b7660531
Oct 30 21:47:01.831613 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:01.831581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:02.109726 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:02.109644    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4" event={"ID":"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636","Type":"ContainerStarted","Data":"26d1a16f13073cb57aa8a693b5bd87c1bedc6e607056e71783c577c8b7660531"}
Oct 30 21:47:03.944404 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:03.944305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:04.233697 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:04.233610    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:04.474050 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:04.474010    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:05.080055 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.079215    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"]
Oct 30 21:47:05.089097 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.089072    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.143130 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.143088    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s9vps\""
Oct 30 21:47:05.144293 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.144264    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"]
Oct 30 21:47:05.198562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.198411    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.198562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.198524    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xm4x\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-kube-api-access-9xm4x\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.299972 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.299939    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xm4x\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-kube-api-access-9xm4x\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.300144 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.300031    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.360181 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.360108    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.360649 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.360631    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xm4x\" (UniqueName: \"kubernetes.io/projected/c23ca503-ced5-42c7-aca6-29ed434afe2d-kube-api-access-9xm4x\") pod \"cert-manager-webhook-5dbbbf75cc-jsjp8\" (UID: \"c23ca503-ced5-42c7-aca6-29ed434afe2d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.407304 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.407239    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:05.451408 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.451374    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:05.646837 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.646803    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"]
Oct 30 21:47:05.658514 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.658491    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.704436 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.702250    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"]
Oct 30 21:47:05.710426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.710395    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:47:05.710645 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.710626    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:47:05.784957 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.784925    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:47:05.785118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.784995    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:47:05.785118 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.785039    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:47:05.785200 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.785118    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7pn2f\""
Oct 30 21:47:05.805426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.805396    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"]
Oct 30 21:47:05.805724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.805698    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.805815 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.805750    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78g7\" (UniqueName: \"kubernetes.io/projected/80851c8d-d65f-440e-a9cc-a9187435f015-kube-api-access-n78g7\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.805976 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.805960    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.806064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.805999    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.806064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.806027    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.806231 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.806115    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.809270 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:05.809234    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc23ca503_ced5_42c7_aca6_29ed434afe2d.slice/crio-4f197339b834cd892a31c98403889cd06f49f748a649bdf8577029185a333522 WatchSource:0}: Error finding container 4f197339b834cd892a31c98403889cd06f49f748a649bdf8577029185a333522: Status 404 returned error can't find the container with id 4f197339b834cd892a31c98403889cd06f49f748a649bdf8577029185a333522
Oct 30 21:47:05.825869 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.825835    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:05.907388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907313    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907354    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907383    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907662 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907418    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907662 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907466    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78g7\" (UniqueName: \"kubernetes.io/projected/80851c8d-d65f-440e-a9cc-a9187435f015-kube-api-access-n78g7\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907662 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907528    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.907844 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.907761    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.910253 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.910224    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80851c8d-d65f-440e-a9cc-a9187435f015-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.910502 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.910484    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.910594 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.910566    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.911884 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.911861    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80851c8d-d65f-440e-a9cc-a9187435f015-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.969651 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.969606    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78g7\" (UniqueName: \"kubernetes.io/projected/80851c8d-d65f-440e-a9cc-a9187435f015-kube-api-access-n78g7\") pod \"opendatahub-operator-controller-manager-578f754bb8-9qfl4\" (UID: \"80851c8d-d65f-440e-a9cc-a9187435f015\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:05.980343 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:05.980317    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:06.028112 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:06.028073    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.129848 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:06.129811    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8" event={"ID":"c23ca503-ced5-42c7-aca6-29ed434afe2d","Type":"ContainerStarted","Data":"4f197339b834cd892a31c98403889cd06f49f748a649bdf8577029185a333522"}
Oct 30 21:47:06.248393 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:06.248305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.482245 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:06.482205    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"]
Oct 30 21:47:06.487241 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:06.487206    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80851c8d_d65f_440e_a9cc_a9187435f015.slice/crio-94bdf59b40f8b8fae2ea6fc1c7a2376df20ea8279ff8902d090e410f50edeb10 WatchSource:0}: Error finding container 94bdf59b40f8b8fae2ea6fc1c7a2376df20ea8279ff8902d090e410f50edeb10: Status 404 returned error can't find the container with id 94bdf59b40f8b8fae2ea6fc1c7a2376df20ea8279ff8902d090e410f50edeb10
Oct 30 21:47:07.135708 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:07.135649    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4" event={"ID":"89b4de6d-c0d7-4a35-a83f-1ce3a61b6636","Type":"ContainerStarted","Data":"8a6e660f55b245d9c5466392f6eb777a0123d6fc1b819c5251fe3da77492f904"}
Oct 30 21:47:07.137274 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:07.137238    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4" event={"ID":"80851c8d-d65f-440e-a9cc-a9187435f015","Type":"ContainerStarted","Data":"94bdf59b40f8b8fae2ea6fc1c7a2376df20ea8279ff8902d090e410f50edeb10"}
Oct 30 21:47:07.252643 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:07.252530    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jrmz4" podStartSLOduration=1.142128166 podStartE2EDuration="6.252512001s" totalImagesPullingTime="5.110383835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.252067161 +0000 UTC m=+280.340804965" watchObservedRunningTime="2026-10-30 21:47:07.252512001 +0000 UTC m=+280.341249798"
Oct 30 21:47:08.047793 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.047043    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.146229 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.146197    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8" event={"ID":"c23ca503-ced5-42c7-aca6-29ed434afe2d","Type":"ContainerStarted","Data":"84a30d7d250d31713c1d6176e91da75ef358083f9a82be02a926e5130350345f"}
Oct 30 21:47:08.146671 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.146265    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:08.264430 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.263628    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8" podStartSLOduration=1.19354563 podStartE2EDuration="3.263609521s" totalImagesPullingTime="2.070063891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:08.263021096 +0000 UTC m=+281.351758899" watchObservedRunningTime="2026-10-30 21:47:08.263609521 +0000 UTC m=+281.352347324"
Oct 30 21:47:08.304991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.304955    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.743693 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.743652    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.991947 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:08.991911    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:09.151501 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:09.151465    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz" event={"ID":"746521d2-0f3e-4cd3-bc9f-0389bf2ea65d","Type":"ContainerStarted","Data":"463828fcf19a3c558d0c3e34b8b311ceb39554b39f414e6c70adc7233c760bc9"}
Oct 30 21:47:09.238568 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:09.238533    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:09.266080 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:09.266016    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4lgz" podStartSLOduration=2.628532483 podStartE2EDuration="10.265997637s" totalImagesPullingTime="7.637465154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:09.264527356 +0000 UTC m=+282.353265184" watchObservedRunningTime="2026-10-30 21:47:09.265997637 +0000 UTC m=+282.354735443"
Oct 30 21:47:09.418098 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:09.418017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:10.157060 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:10.157030    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4" event={"ID":"80851c8d-d65f-440e-a9cc-a9187435f015","Type":"ContainerStarted","Data":"b94056f6d775113e4c97d33cfd991497bac53cf132e93637d14bcfbe80b0d5d9"}
Oct 30 21:47:10.374538 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:10.374464    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:10.609542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:10.609507    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:10.971603 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:10.971569    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:11.162430 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:11.162393    4410 generic.go:356] "Generic (PLEG): container finished" podID="80851c8d-d65f-440e-a9cc-a9187435f015" containerID="b94056f6d775113e4c97d33cfd991497bac53cf132e93637d14bcfbe80b0d5d9" exitCode=0
Oct 30 21:47:11.162845 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:11.162475    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4" event={"ID":"80851c8d-d65f-440e-a9cc-a9187435f015","Type":"ContainerDied","Data":"b94056f6d775113e4c97d33cfd991497bac53cf132e93637d14bcfbe80b0d5d9"}
Oct 30 21:47:11.169649 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:11.169613    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:11.607501 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:11.607465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:11.845295 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:11.845260    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:12.168236 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:12.168139    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4" event={"ID":"80851c8d-d65f-440e-a9cc-a9187435f015","Type":"ContainerStarted","Data":"c5459ca75c568d379ea6924cd32bdca035a77f1e74e29e9dbc1fdc94ee91c647"}
Oct 30 21:47:12.168672 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:12.168258    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:12.306366 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:12.306310    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4" podStartSLOduration=2.537328838 podStartE2EDuration="7.306297207s" totalImagesPullingTime="4.334445225s" totalInitContainerRuntime="434.523144ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:12.304135204 +0000 UTC m=+285.392873008" watchObservedRunningTime="2026-10-30 21:47:12.306297207 +0000 UTC m=+285.395035010"
Oct 30 21:47:14.847694 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:14.847661    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:15.158912 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:15.158888    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jsjp8"
Oct 30 21:47:19.056386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:19.056350    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:19.254269 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:19.254223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:20.519982 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:47:20.519945    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:47:23.174733 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:23.174705    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-9qfl4"
Oct 30 21:47:24.366636 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:24.366601    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:24.528919 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:24.528876    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:24.760267 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:24.760230    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:27.433629 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:27.433590    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:27.615417 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:27.615386    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:33.175385 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:33.175353    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:33.362701 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:33.362668    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:33.680133 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:33.680103    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.460539 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.460505    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.556793 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.556749    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"]
Oct 30 21:47:35.578560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.578536    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.618016 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.617990    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"]
Oct 30 21:47:35.630750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.630707    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:47:35.652286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.652260    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc23021-e428-442f-b8e1-093e06020210-tls-certs\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.652402 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.652313    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc23021-e428-442f-b8e1-093e06020210-tmp\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.652402 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.652334    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6d9p\" (UniqueName: \"kubernetes.io/projected/0bc23021-e428-442f-b8e1-093e06020210-kube-api-access-q6d9p\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.665308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.665279    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.665888 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.665869    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:47:35.666005 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.665905    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:47:35.666005 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.665985    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:47:35.679693 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.679675    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vrvww\""
Oct 30 21:47:35.753569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.753489    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc23021-e428-442f-b8e1-093e06020210-tls-certs\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.753569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.753536    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc23021-e428-442f-b8e1-093e06020210-tmp\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.753569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.753555    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6d9p\" (UniqueName: \"kubernetes.io/projected/0bc23021-e428-442f-b8e1-093e06020210-kube-api-access-q6d9p\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.755763 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.755737    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc23021-e428-442f-b8e1-093e06020210-tmp\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.755991 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.755973    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc23021-e428-442f-b8e1-093e06020210-tls-certs\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.812703 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.812669    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6d9p\" (UniqueName: \"kubernetes.io/projected/0bc23021-e428-442f-b8e1-093e06020210-kube-api-access-q6d9p\") pod \"kube-auth-proxy-76f5684988-h5t9m\" (UID: \"0bc23021-e428-442f-b8e1-093e06020210\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:35.844828 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.844797    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.888673 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:35.888647    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"
Oct 30 21:47:36.019861 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.019760    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:36.195253 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.195214    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:36.256997 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.256967    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-h5t9m"]
Oct 30 21:47:36.259839 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:36.259810    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc23021_e428_442f_b8e1_093e06020210.slice/crio-647dd3f60e5ce03a0d50bce09a8a5dfc097d4b41a5917f7940e24877bf85cf06 WatchSource:0}: Error finding container 647dd3f60e5ce03a0d50bce09a8a5dfc097d4b41a5917f7940e24877bf85cf06: Status 404 returned error can't find the container with id 647dd3f60e5ce03a0d50bce09a8a5dfc097d4b41a5917f7940e24877bf85cf06
Oct 30 21:47:36.261987 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.261964    4410 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:47:36.266154 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.266133    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m" event={"ID":"0bc23021-e428-442f-b8e1-093e06020210","Type":"ContainerStarted","Data":"647dd3f60e5ce03a0d50bce09a8a5dfc097d4b41a5917f7940e24877bf85cf06"}
Oct 30 21:47:36.364141 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.364089    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:36.536531 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.536497    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:36.759225 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:36.759192    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:38.514393 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.514345    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"]
Oct 30 21:47:38.521537 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.521515    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.576050 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.576017    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"]
Oct 30 21:47:38.579604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.579569    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11c68a2c-04e8-4ac7-874c-6baf801a2607-manager-config\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.579751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.579686    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-cert\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.579751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.579730    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62jz\" (UniqueName: \"kubernetes.io/projected/11c68a2c-04e8-4ac7-874c-6baf801a2607-kube-api-access-w62jz\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.579927 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.579898    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-metrics-certs\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.623343 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.623265    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:47:38.623517 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.623496    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6757d\""
Oct 30 21:47:38.623682 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.623666    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:47:38.623793 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.623683    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:47:38.680607 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.680562    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-cert\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.680607 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.680606    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62jz\" (UniqueName: \"kubernetes.io/projected/11c68a2c-04e8-4ac7-874c-6baf801a2607-kube-api-access-w62jz\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.680885 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.680694    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-metrics-certs\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.680885 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.680761    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11c68a2c-04e8-4ac7-874c-6baf801a2607-manager-config\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.681404 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.681381    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11c68a2c-04e8-4ac7-874c-6baf801a2607-manager-config\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.683805 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.683757    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-cert\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.684605 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.684578    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c68a2c-04e8-4ac7-874c-6baf801a2607-metrics-certs\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.738530 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.738501    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62jz\" (UniqueName: \"kubernetes.io/projected/11c68a2c-04e8-4ac7-874c-6baf801a2607-kube-api-access-w62jz\") pod \"jobset-controller-manager-848d766f4b-729lf\" (UID: \"11c68a2c-04e8-4ac7-874c-6baf801a2607\") " pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:38.835091 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:38.835003    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:39.214956 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:39.214922    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"]
Oct 30 21:47:39.217921 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:47:39.217894    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c68a2c_04e8_4ac7_874c_6baf801a2607.slice/crio-5a597ba3adbf90219a5a4d366b3b142a836f5bd19819f52e35c26bcc043f38e6 WatchSource:0}: Error finding container 5a597ba3adbf90219a5a4d366b3b142a836f5bd19819f52e35c26bcc043f38e6: Status 404 returned error can't find the container with id 5a597ba3adbf90219a5a4d366b3b142a836f5bd19819f52e35c26bcc043f38e6
Oct 30 21:47:39.281172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:39.281140    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m" event={"ID":"0bc23021-e428-442f-b8e1-093e06020210","Type":"ContainerStarted","Data":"d8a5887fd550968d47073e6e3a8dcdb02a88ec9b39aa7cafa46aa92bfdd7bc41"}
Oct 30 21:47:39.282352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:39.282332    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf" event={"ID":"11c68a2c-04e8-4ac7-874c-6baf801a2607","Type":"ContainerStarted","Data":"5a597ba3adbf90219a5a4d366b3b142a836f5bd19819f52e35c26bcc043f38e6"}
Oct 30 21:47:39.396658 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:39.396609    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76f5684988-h5t9m" podStartSLOduration=1.7325032550000001 podStartE2EDuration="4.396594508s" totalImagesPullingTime="2.664091253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:39.395300234 +0000 UTC m=+312.484038036" watchObservedRunningTime="2026-10-30 21:47:39.396594508 +0000 UTC m=+312.485332311"
Oct 30 21:47:40.238414 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:40.238328    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:40.569790 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:40.569691    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:40.787119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:40.787085    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:42.038362 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:42.038330    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:42.300654 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:42.300586    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:43.302725 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:43.302686    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf" event={"ID":"11c68a2c-04e8-4ac7-874c-6baf801a2607","Type":"ContainerStarted","Data":"92c3c41c317296f6df90b5db8b95808c2605f4b4c0ec6f8ec52aab31b88a9d39"}
Oct 30 21:47:43.303129 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:43.302833    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:47:43.337585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:43.337554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:43.420165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:43.420107    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf" podStartSLOduration=1.752274007 podStartE2EDuration="5.420092898s" totalImagesPullingTime="3.667818891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:43.419035625 +0000 UTC m=+316.507773430" watchObservedRunningTime="2026-10-30 21:47:43.420092898 +0000 UTC m=+316.508830700"
Oct 30 21:47:50.521480 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:47:50.521435    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:47:54.311899 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:47:54.311870    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-848d766f4b-729lf"
Oct 30 21:48:05.716280 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:05.716241    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:05.940950 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:05.940912    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:06.409674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:06.409626    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:15.184723 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:15.184684    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:20.521822 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:48:20.521767    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:48:37.839109 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:37.839045    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:38.039660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:38.039554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:38.301678 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:38.301641    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:38.762474 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:38.762440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:45.470483 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:45.470424    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:45.649279 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:45.649226    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:50.141377 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:50.141345    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:50.310712 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:50.310656    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:50.521057 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:48:50.520971    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:48:52.323666 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:52.323623    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:52.478977 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:52.478941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:52.626247 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:52.626160    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:52.773250 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:52.773205    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:52.922963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:52.922932    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.122611 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.122583    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.273071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.272994    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.305886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.305856    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:48:53.308261 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.308238    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.397516 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.397487    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98x6q\" (UniqueName: \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.397890 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.397535    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.409217 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.409190    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:48:53.409427 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.409407    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:48:53.409523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.409433    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 21:48:53.409523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.409458    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-s9pzz\""
Oct 30 21:48:53.410978 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.410960    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:48:53.464794 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.464742    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.498900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.498868    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98x6q\" (UniqueName: \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.499064 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.498919    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.501258 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.501233    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.535037 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.534969    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-l88st"]
Oct 30 21:48:53.537166 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.537151    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.558623 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.558599    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98x6q\" (UniqueName: \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\") pod \"llmisvc-controller-manager-869d5dcc5d-wvwtp\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.589176 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.589156    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:48:53.589272 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.589232    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:48:53.589320 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.589274    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7snqn\""
Oct 30 21:48:53.595575 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.595548    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-l88st"]
Oct 30 21:48:53.599811 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.599785    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-metrics-certs\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.599915 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.599818    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-cert\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.599915 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.599840    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxlj\" (UniqueName: \"kubernetes.io/projected/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-kube-api-access-xhxlj\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.618753 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.618732    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:53.701270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.701239    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-metrics-certs\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.701395 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.701316    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-cert\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.701395 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.701341    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxlj\" (UniqueName: \"kubernetes.io/projected/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-kube-api-access-xhxlj\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.703544 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.703522    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-cert\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.703642 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.703586    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-metrics-certs\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.757363 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.757340    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxlj\" (UniqueName: \"kubernetes.io/projected/b0ecfeba-6271-47aa-8025-1cfd2b5f4e53-kube-api-access-xhxlj\") pod \"odh-model-controller-769875db56-l88st\" (UID: \"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53\") " pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.849080 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.849030    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:53.988203 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:53.988172    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:48:53.990975 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:48:53.990950    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod419caa46_f51f_4d1f_ae47_32bcdf60221c.slice/crio-e2ef02496b5ecc03eb0106031080cd971f62a5e6fd1aa50bd4204cc3bc079cbb WatchSource:0}: Error finding container e2ef02496b5ecc03eb0106031080cd971f62a5e6fd1aa50bd4204cc3bc079cbb: Status 404 returned error can't find the container with id e2ef02496b5ecc03eb0106031080cd971f62a5e6fd1aa50bd4204cc3bc079cbb
Oct 30 21:48:54.214091 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:54.214004    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-l88st"]
Oct 30 21:48:54.216529 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:48:54.216508    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ecfeba_6271_47aa_8025_1cfd2b5f4e53.slice/crio-f6db993e0ee9372028d77f9f14fdc294e8dfadd4b05caf94ddfce013accd336c WatchSource:0}: Error finding container f6db993e0ee9372028d77f9f14fdc294e8dfadd4b05caf94ddfce013accd336c: Status 404 returned error can't find the container with id f6db993e0ee9372028d77f9f14fdc294e8dfadd4b05caf94ddfce013accd336c
Oct 30 21:48:54.593333 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:54.593257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.594143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:54.594116    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-l88st" event={"ID":"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53","Type":"ContainerStarted","Data":"f6db993e0ee9372028d77f9f14fdc294e8dfadd4b05caf94ddfce013accd336c"}
Oct 30 21:48:54.595371 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:54.595350    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" event={"ID":"419caa46-f51f-4d1f-ae47-32bcdf60221c","Type":"ContainerStarted","Data":"e2ef02496b5ecc03eb0106031080cd971f62a5e6fd1aa50bd4204cc3bc079cbb"}
Oct 30 21:48:56.607640 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:56.607548    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" event={"ID":"419caa46-f51f-4d1f-ae47-32bcdf60221c","Type":"ContainerStarted","Data":"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"}
Oct 30 21:48:56.608065 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:56.607656    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:48:56.722182 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:56.722129    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" podStartSLOduration=1.5122241669999998 podStartE2EDuration="3.722111435s" totalImagesPullingTime="2.209887268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:56.72050775 +0000 UTC m=+389.809245554" watchObservedRunningTime="2026-10-30 21:48:56.722111435 +0000 UTC m=+389.810849242"
Oct 30 21:48:57.612982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:57.612939    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-l88st" event={"ID":"b0ecfeba-6271-47aa-8025-1cfd2b5f4e53","Type":"ContainerStarted","Data":"c8c0ee67fda1194ccdc141bc5624a24b2672d06e48832ac4c689400bf446fea0"}
Oct 30 21:48:57.613364 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:57.613145    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:48:57.728805 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:57.728735    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-l88st" podStartSLOduration=1.557149913 podStartE2EDuration="4.728719744s" totalImagesPullingTime="3.171569831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:57.727681706 +0000 UTC m=+390.816419523" watchObservedRunningTime="2026-10-30 21:48:57.728719744 +0000 UTC m=+390.817457583"
Oct 30 21:48:59.686434 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:59.686392    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:59.895990 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:48:59.895957    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.046113 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.046024    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.207063 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.207024    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.361116 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.361038    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.519003 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.518967    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.666900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.666869    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.822679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.822631    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:00.973632 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:00.973544    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:08.619883 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:08.619846    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-l88st"
Oct 30 21:49:15.504344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:15.504257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:19.137373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:19.137329    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:19.310266 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:19.310223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:19.848522 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:19.848477    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:20.521366 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:49:20.521326    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:49:21.338603 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:21.338569    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:21.498311 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:21.498277    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:21.655520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:21.655478    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:21.823669 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:21.823634    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:27.615018 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:27.614985    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:49:35.715033 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:35.714990    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:35.967454 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:49:35.967375    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:50.521543 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:49:50.521509    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:50:01.449923 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.449892    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"]
Oct 30 21:50:01.452681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.452665    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.512930 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.512897    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"]
Oct 30 21:50:01.548634 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.548609    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a508794c-35dd-46d6-b7c3-fb2ea20481cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.548783 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.548646    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll8t\" (UniqueName: \"kubernetes.io/projected/a508794c-35dd-46d6-b7c3-fb2ea20481cc-kube-api-access-mll8t\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.649088 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.649055    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a508794c-35dd-46d6-b7c3-fb2ea20481cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.649231 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.649099    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll8t\" (UniqueName: \"kubernetes.io/projected/a508794c-35dd-46d6-b7c3-fb2ea20481cc-kube-api-access-mll8t\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.651474 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.651451    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a508794c-35dd-46d6-b7c3-fb2ea20481cc-cert\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.705293 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.705229    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll8t\" (UniqueName: \"kubernetes.io/projected/a508794c-35dd-46d6-b7c3-fb2ea20481cc-kube-api-access-mll8t\") pod \"llmisvc-controller-manager-6466c554d8-w6rkh\" (UID: \"a508794c-35dd-46d6-b7c3-fb2ea20481cc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:01.765258 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:01.765220    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:02.134077 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:02.133912    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"]
Oct 30 21:50:02.138894 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:50:02.138861    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda508794c_35dd_46d6_b7c3_fb2ea20481cc.slice/crio-93225d573f00f8255f0d12d3f4bfe7bf8bd54dc96c936af813ce1fa314200892 WatchSource:0}: Error finding container 93225d573f00f8255f0d12d3f4bfe7bf8bd54dc96c936af813ce1fa314200892: Status 404 returned error can't find the container with id 93225d573f00f8255f0d12d3f4bfe7bf8bd54dc96c936af813ce1fa314200892
Oct 30 21:50:02.881015 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:02.880985    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh" event={"ID":"a508794c-35dd-46d6-b7c3-fb2ea20481cc","Type":"ContainerStarted","Data":"8d224a720740f71af74a4d6543017c671b462b845e8e86ef18ff0972a6b3284c"}
Oct 30 21:50:02.881305 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:02.881045    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh" event={"ID":"a508794c-35dd-46d6-b7c3-fb2ea20481cc","Type":"ContainerStarted","Data":"93225d573f00f8255f0d12d3f4bfe7bf8bd54dc96c936af813ce1fa314200892"}
Oct 30 21:50:02.881305 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:02.881139    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:02.993690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:02.993639    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh" podStartSLOduration=1.316471703 podStartE2EDuration="1.993625023s" totalImagesPullingTime="677.15332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:02.99323663 +0000 UTC m=+456.081974432" watchObservedRunningTime="2026-10-30 21:50:02.993625023 +0000 UTC m=+456.082362825"
Oct 30 21:50:07.903199 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:07.903159    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:08.126023 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:08.125992    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:15.844503 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:15.844465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:20.520501 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:50:20.520464    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:50:33.887253 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:33.887224    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-w6rkh"
Oct 30 21:50:34.042263 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.042227    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:50:34.042649 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.042599    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" podUID="419caa46-f51f-4d1f-ae47-32bcdf60221c" containerName="manager" containerID="cri-o://24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d" gracePeriod=30
Oct 30 21:50:34.282946 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.282921    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:50:34.303063 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.303032    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\" (UniqueName: \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\") pod \"419caa46-f51f-4d1f-ae47-32bcdf60221c\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") "
Oct 30 21:50:34.303164 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.303072    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\" (UniqueName: \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\") pod \"419caa46-f51f-4d1f-ae47-32bcdf60221c\" (UID: \"419caa46-f51f-4d1f-ae47-32bcdf60221c\") "
Oct 30 21:50:34.305054 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.305033    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q" pod "419caa46-f51f-4d1f-ae47-32bcdf60221c" (UID: "419caa46-f51f-4d1f-ae47-32bcdf60221c"). InnerVolumeSpecName "kube-api-access-98x6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:34.305138 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.305122    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert" pod "419caa46-f51f-4d1f-ae47-32bcdf60221c" (UID: "419caa46-f51f-4d1f-ae47-32bcdf60221c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:34.404262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.404233    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98x6q\" (UniqueName: \"kubernetes.io/projected/419caa46-f51f-4d1f-ae47-32bcdf60221c-kube-api-access-98x6q\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:34.404262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:34.404258    4410 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/419caa46-f51f-4d1f-ae47-32bcdf60221c-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.011286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.011253    4410 generic.go:356] "Generic (PLEG): container finished" podID="419caa46-f51f-4d1f-ae47-32bcdf60221c" containerID="24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d" exitCode=0
Oct 30 21:50:35.011720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.011301    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"
Oct 30 21:50:35.011720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.011343    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" event={"ID":"419caa46-f51f-4d1f-ae47-32bcdf60221c","Type":"ContainerDied","Data":"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"}
Oct 30 21:50:35.011720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.011396    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp" event={"ID":"419caa46-f51f-4d1f-ae47-32bcdf60221c","Type":"ContainerDied","Data":"e2ef02496b5ecc03eb0106031080cd971f62a5e6fd1aa50bd4204cc3bc079cbb"}
Oct 30 21:50:35.011720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.011420    4410 scope.go:122] "RemoveContainer" containerID="24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"
Oct 30 21:50:35.013165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.013137    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:35.021319 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.021282    4410 scope.go:122] "RemoveContainer" containerID="24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"
Oct 30 21:50:35.021676 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:50:35.021607    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d\": container with ID starting with 24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d not found: ID does not exist" containerID="24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"
Oct 30 21:50:35.021676 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.021647    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d"} err="failed to get container status \"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d\": rpc error: code = NotFound desc = could not find container \"24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d\": container with ID starting with 24dbb526a803ba86ca23123b32b89808ebedd4c820725b664b0eade0d483aa4d not found: ID does not exist"
Oct 30 21:50:35.172353 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.172324    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:50:35.175800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.175764    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wvwtp"]
Oct 30 21:50:35.181266 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.181237    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:35.443492 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:35.443463    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:37.017403 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:37.017375    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419caa46-f51f-4d1f-ae47-32bcdf60221c" path="/var/lib/kubelet/pods/419caa46-f51f-4d1f-ae47-32bcdf60221c/volumes"
Oct 30 21:50:42.469061 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:42.468981    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:42.673696 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:42.673649    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:50.521990 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:50:50.521953    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:50:57.928605 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:57.928565    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:58.097832 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:50:58.097798    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:01.972004 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:01.971970    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:16.177425 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:16.177373    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:16.335153 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:16.335102    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:16.761010 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:16.760965    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:17.940755 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:17.940719    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:18.164781 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:18.164753    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:20.520969 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:51:20.520931    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:51:24.842481 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:24.842449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:26.619481 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:26.619447    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:26.793709 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:26.793664    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:32.457214 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:32.457173    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:35.920276 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:35.920238    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:39.045196 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:39.045159    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:42.465881 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:42.465846    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:46.212052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:46.212001    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:49.633144 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:49.633109    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:50.520575 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:51:50.520536    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:51:53.072220 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:53.072179    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:58.361210 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:51:58.361166    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:09.443188 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.443153    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 21:52:09.443617 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.443483    4410 cpu_manager.go:414] "containerMap: removing container" podUID="419caa46-f51f-4d1f-ae47-32bcdf60221c" containerName="manager"
Oct 30 21:52:09.443617 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.443493    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419caa46-f51f-4d1f-ae47-32bcdf60221c" containerName="manager"
Oct 30 21:52:09.443617 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.443575    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="419caa46-f51f-4d1f-ae47-32bcdf60221c" containerName="manager"
Oct 30 21:52:09.446588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.446571    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.498148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.498122    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dz4rf\""
Oct 30 21:52:09.547525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.547501    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:52:09.585686 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.585662    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 21:52:09.591688 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591660    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.591817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591715    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.591875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591812    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.591875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591846    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6q2\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.591959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591885    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.591959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.591912    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.592068 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.592030    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.592122 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.592103    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.592180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.592137    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.692791 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.692736    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.692791 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.692785    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693031 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.692898    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693031 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.692947    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6q2\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693031 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.692976    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693031 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693002    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693040    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693126    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693178    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693218    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693226    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693557 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693529    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693687 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693563    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.693843 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.693824    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.695081 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.695061    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.695406 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.695389    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.749627 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.749598    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.749740 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.749687    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6q2\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\") pod \"maas-default-gateway-openshift-default-694db55c87-fjr78\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:09.758545 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:09.758524    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:10.180020 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:10.179964    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 21:52:10.396340 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:10.396305    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" event={"ID":"0cbeca0a-5b04-4bbc-b01d-3f693635106a","Type":"ContainerStarted","Data":"bc0f0c8a4903c36430afb8523551b7a0417a3a407f70a11d74d07557e40fc298"}
Oct 30 21:52:14.602508 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:14.602476    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:14.602792 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:14.602560    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:14.602792 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:14.602601    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:15.421688 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:15.421654    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" event={"ID":"0cbeca0a-5b04-4bbc-b01d-3f693635106a","Type":"ContainerStarted","Data":"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"}
Oct 30 21:52:15.588719 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:15.588665    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podStartSLOduration=2.173220909 podStartE2EDuration="6.588647532s" totalImagesPullingTime="4.415426623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:15.587763249 +0000 UTC m=+588.676501077" watchObservedRunningTime="2026-10-30 21:52:15.588647532 +0000 UTC m=+588.677385340"
Oct 30 21:52:15.766853 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:15.765053    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:15.767320 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:15.767108    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:15.767320 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:15.767177    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:16.510671 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:16.510568    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:16.763237 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:16.763144    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:16.763525 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:16.763486    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:17.762798 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:17.761863    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:17.762798 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:17.761936    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:18.767131 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:18.767096    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:18.767749 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:18.767716    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:19.759205 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:19.759162    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:19.759610 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:19.759576    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:20.522047 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:52:20.522008    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:52:20.758881 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:20.758851    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:20.759037 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:20.758909    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:21.759787 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:21.759749    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 21:52:21.760228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:21.759826    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 21:52:22.210340 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:22.210303    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:22.418320 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:22.418262    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:22.927266 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:22.927233    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:22.927797 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:22.927757    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:22.928599 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:22.928580    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 21:52:23.834085 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:23.834048    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:24.096088 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.096003    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:24.579131 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.579100    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 21:52:24.582738 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.582715    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.643943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.643910    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.643943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.643947    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.643966    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644033    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644075    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644109    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn5jp\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644143    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644215    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.644482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.644253    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.685651 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.685623    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 21:52:24.745461 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745433    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745491    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745508    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745522    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745538    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745558    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745587    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5jp\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745618    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745647    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.745944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.745908    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.746228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.746196    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.746332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.746238    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.746332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.746295    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.746332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.746324    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.748013 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.747989    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.748283 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.748264    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.804291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.804256    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.804520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.804451    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5jp\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\") pod \"maas-default-gateway-openshift-default-694db55c87-r6nlx\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:24.897977 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:24.897954    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:25.321913 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.321831    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 21:52:25.324713 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:52:25.324681    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494e40ea_8a23_4189_8191_0035bf7ed04f.slice/crio-5a32646aa57d7cccfb14ac19f8bb5eae14631af2c8de4fe3376768c23fa0b34a WatchSource:0}: Error finding container 5a32646aa57d7cccfb14ac19f8bb5eae14631af2c8de4fe3376768c23fa0b34a: Status 404 returned error can't find the container with id 5a32646aa57d7cccfb14ac19f8bb5eae14631af2c8de4fe3376768c23fa0b34a
Oct 30 21:52:25.327738 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.327707    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:25.327831 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.327792    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:25.327831 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.327822    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:25.471223 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.471184    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" event={"ID":"494e40ea-8a23-4189-8191-0035bf7ed04f","Type":"ContainerStarted","Data":"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"}
Oct 30 21:52:25.471373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.471250    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" event={"ID":"494e40ea-8a23-4189-8191-0035bf7ed04f","Type":"ContainerStarted","Data":"5a32646aa57d7cccfb14ac19f8bb5eae14631af2c8de4fe3376768c23fa0b34a"}
Oct 30 21:52:25.596218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.596119    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podStartSLOduration=1.596107339 podStartE2EDuration="1.596107339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:25.59538775 +0000 UTC m=+598.684125565" watchObservedRunningTime="2026-10-30 21:52:25.596107339 +0000 UTC m=+598.684845142"
Oct 30 21:52:25.898303 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:25.898258    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:26.900559 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:26.900524    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:26.901006 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:26.900577    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:27.898746 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:27.898712    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:27.898917 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:27.898786    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:28.899196 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:28.899161    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:28.899572 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:28.899216    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:29.898507 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:29.898467    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:29.898695 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:29.898538    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:30.898558 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:30.898524    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:30.898961 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:30.898582    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:31.899073 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:31.899041    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:52:31.899452 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:31.899094    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:52:33.589286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:33.589252    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:33.589731 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:33.589699    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:33.590468 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:33.590443    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 21:52:50.521107 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:52:50.521066    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:52:57.038296 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:52:57.038266    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.069344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:17.069312    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:20.521581 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:53:20.521545    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:53:22.761582 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:22.761545    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.070958 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:23.070874    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.381035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:23.380957    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.717874 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:23.717832    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.981959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:23.981881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.458797 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:24.458743    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.684707 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:24.684673    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:25.462916 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:25.462881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:25.663488 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:25.663453    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.903751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:26.903715    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:27.190287 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:27.190206    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:27.906286 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:27.906254    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.142212 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:28.142179    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:30.442596 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:30.442564    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:30.618944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:30.618908    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:31.204421 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:31.204382    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:31.366962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:31.366926    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:31.523786 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:31.523701    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:32.471940 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:32.471908    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:32.680455 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:32.680417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.224851 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:33.224820    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.460814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:33.460760    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.628760 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:33.628659    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.786262 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:33.786222    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.951817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:33.951759    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.376946 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:35.376912    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:35.545690 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:35.545643    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.009731 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:38.009698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.204452 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:38.204406    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.618750 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:38.618704    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.829510 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:38.829472    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.712482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:39.712443    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.932352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:39.932248    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:40.107291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:40.107214    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:44.112702 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:44.112663    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:47.599649 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:47.599601    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:47.779482 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:47.779449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:49.019462 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:49.019426    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:49.199076 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:49.199046    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:49.535789 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:49.535748    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:50.522070 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:53:50.522037    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:53:50.690562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:50.690528    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:51.292605 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:51.292567    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:54.694098 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:54.694055    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:55.103569 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:55.103491    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:55.336995 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:55.336962    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:59.644637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:59.644597    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:59.857963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:53:59.857930    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:00.017950 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:00.017862    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:00.187035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:00.187003    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:05.374346 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:05.374308    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.496171 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:10.496137    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.279496 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.279461    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"]
Oct 30 21:54:13.282713 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.282696    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.377038 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.377009    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.377159 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.377075    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.377159 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.377152    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhgc\" (UniqueName: \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.384107 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.384076    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:13.384239 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.384122    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:13.384239 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.384128    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:54:13.384239 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.384191    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:13.385617 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.385598    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"]
Oct 30 21:54:13.477703 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.477673    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhgc\" (UniqueName: \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.477857 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.477726    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.477857 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.477821    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.478371 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.478355    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.480068 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.480046    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.535128 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.535076    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhgc\" (UniqueName: \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\") pod \"maas-api-revoke-keys-e2e-migrate-c6317794-llwgp\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.595140 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.595112    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:13.961070 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.961049    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"]
Oct 30 21:54:13.963590 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:54:13.963565    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaff3b3a_53b1_4689_8492_c2c045db1b18.slice/crio-22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e WatchSource:0}: Error finding container 22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e: Status 404 returned error can't find the container with id 22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e
Oct 30 21:54:13.965865 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:13.965850    4410 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:54:14.956312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:14.956278    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp" event={"ID":"daff3b3a-53b1-4689-8492-c2c045db1b18","Type":"ContainerStarted","Data":"22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e"}
Oct 30 21:54:15.796312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:15.796279    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:17.227174 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:17.227133    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:18.976867 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:18.976791    4410 generic.go:356] "Generic (PLEG): container finished" podID="daff3b3a-53b1-4689-8492-c2c045db1b18" containerID="bed5ef76ac1b65367aea87421b7263da45ada7f574c823aa2c1f92153b90414b" exitCode=0
Oct 30 21:54:18.977220 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:18.976878    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp" event={"ID":"daff3b3a-53b1-4689-8492-c2c045db1b18","Type":"ContainerDied","Data":"bed5ef76ac1b65367aea87421b7263da45ada7f574c823aa2c1f92153b90414b"}
Oct 30 21:54:20.115062 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.115039    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:20.237101 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.237025    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\") pod \"daff3b3a-53b1-4689-8492-c2c045db1b18\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") "
Oct 30 21:54:20.237235 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.237180    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\") pod \"daff3b3a-53b1-4689-8492-c2c045db1b18\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") "
Oct 30 21:54:20.237235 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.237215    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\" (UniqueName: \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\") pod \"daff3b3a-53b1-4689-8492-c2c045db1b18\" (UID: \"daff3b3a-53b1-4689-8492-c2c045db1b18\") "
Oct 30 21:54:20.237414 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.237389    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca" pod "daff3b3a-53b1-4689-8492-c2c045db1b18" (UID: "daff3b3a-53b1-4689-8492-c2c045db1b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:20.237510 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.237493    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.239429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.239405    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca" pod "daff3b3a-53b1-4689-8492-c2c045db1b18" (UID: "daff3b3a-53b1-4689-8492-c2c045db1b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:20.239429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.239416    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc" pod "daff3b3a-53b1-4689-8492-c2c045db1b18" (UID: "daff3b3a-53b1-4689-8492-c2c045db1b18"). InnerVolumeSpecName "kube-api-access-mvhgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:20.338444 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.338417    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daff3b3a-53b1-4689-8492-c2c045db1b18-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.338444 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.338441    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvhgc\" (UniqueName: \"kubernetes.io/projected/daff3b3a-53b1-4689-8492-c2c045db1b18-kube-api-access-mvhgc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.525647 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:54:20.525555    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:54:20.698583 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.698548    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:20.903273 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.903244    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:20.987837 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.987802    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp" event={"ID":"daff3b3a-53b1-4689-8492-c2c045db1b18","Type":"ContainerDied","Data":"22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e"}
Oct 30 21:54:20.987837 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.987827    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"
Oct 30 21:54:20.988020 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:20.987889    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bb3c679f401e3b8bebc01818b4e0a32212b4b8d2ae09c1809b57c5b8ef134e"
Oct 30 21:54:22.672670 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.672641    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"]
Oct 30 21:54:22.673075 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.673003    4410 cpu_manager.go:414] "containerMap: removing container" podUID="daff3b3a-53b1-4689-8492-c2c045db1b18" containerName="revoke-keys"
Oct 30 21:54:22.673075 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.673015    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daff3b3a-53b1-4689-8492-c2c045db1b18" containerName="revoke-keys"
Oct 30 21:54:22.673148 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.673083    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="daff3b3a-53b1-4689-8492-c2c045db1b18" containerName="revoke-keys"
Oct 30 21:54:22.676150 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.676135    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.761344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.761312    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck8q\" (UniqueName: \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.761344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.761349    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.761593 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.761488    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.778179 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.778147    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:54:22.778179 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.778172    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:22.778379 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.778215    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:22.778379 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.778241    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:22.780186 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.780161    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"]
Oct 30 21:54:22.862485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.862451    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.862668 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.862504    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck8q\" (UniqueName: \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.862668 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.862527    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.863182 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.863156    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.864785 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.864753    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.920421 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.920391    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck8q\" (UniqueName: \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\") pod \"maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:22.988326 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:22.988236    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:23.364458 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:23.364385    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"]
Oct 30 21:54:23.368965 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:54:23.368937    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9722350_cb9b_4475_a6cf_447f9d355941.slice/crio-1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2 WatchSource:0}: Error finding container 1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2: Status 404 returned error can't find the container with id 1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2
Oct 30 21:54:24.003975 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:24.003940    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9" event={"ID":"a9722350-cb9b-4475-a6cf-447f9d355941","Type":"ContainerStarted","Data":"e818788e39ea625fcec9d69dbb58be8c81fdf72d3a0b695618338c6048d5ef5d"}
Oct 30 21:54:24.004435 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:24.003997    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9" event={"ID":"a9722350-cb9b-4475-a6cf-447f9d355941","Type":"ContainerStarted","Data":"1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2"}
Oct 30 21:54:24.127640 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:24.127599    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9" podStartSLOduration=2.127586827 podStartE2EDuration="2.127586827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:24.126834511 +0000 UTC m=+717.215572316" watchObservedRunningTime="2026-10-30 21:54:24.127586827 +0000 UTC m=+717.216324630"
Oct 30 21:54:25.010154 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:25.010115    4410 generic.go:356] "Generic (PLEG): container finished" podID="a9722350-cb9b-4475-a6cf-447f9d355941" containerID="e818788e39ea625fcec9d69dbb58be8c81fdf72d3a0b695618338c6048d5ef5d" exitCode=0
Oct 30 21:54:25.010516 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:25.010162    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9" event={"ID":"a9722350-cb9b-4475-a6cf-447f9d355941","Type":"ContainerDied","Data":"e818788e39ea625fcec9d69dbb58be8c81fdf72d3a0b695618338c6048d5ef5d"}
Oct 30 21:54:25.539044 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:25.539009    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:26.099247 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.099214    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:26.157968 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.157942    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:26.291454 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.291369    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\") pod \"a9722350-cb9b-4475-a6cf-447f9d355941\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") "
Oct 30 21:54:26.291615 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.291448    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\") pod \"a9722350-cb9b-4475-a6cf-447f9d355941\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") "
Oct 30 21:54:26.291615 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.291512    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\" (UniqueName: \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\") pod \"a9722350-cb9b-4475-a6cf-447f9d355941\" (UID: \"a9722350-cb9b-4475-a6cf-447f9d355941\") "
Oct 30 21:54:26.291891 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.291858    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca" pod "a9722350-cb9b-4475-a6cf-447f9d355941" (UID: "a9722350-cb9b-4475-a6cf-447f9d355941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:26.293517 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.293490    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q" pod "a9722350-cb9b-4475-a6cf-447f9d355941" (UID: "a9722350-cb9b-4475-a6cf-447f9d355941"). InnerVolumeSpecName "kube-api-access-sck8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:26.293633 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.293555    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca" pod "a9722350-cb9b-4475-a6cf-447f9d355941" (UID: "a9722350-cb9b-4475-a6cf-447f9d355941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:26.350506 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.350471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:26.392657 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.392628    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:26.392657 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.392652    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9722350-cb9b-4475-a6cf-447f9d355941-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:26.392859 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:26.392665    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sck8q\" (UniqueName: \"kubernetes.io/projected/a9722350-cb9b-4475-a6cf-447f9d355941-kube-api-access-sck8q\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:27.020376 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:27.020344    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9" event={"ID":"a9722350-cb9b-4475-a6cf-447f9d355941","Type":"ContainerDied","Data":"1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2"}
Oct 30 21:54:27.020533 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:27.020381    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"
Oct 30 21:54:27.020533 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:27.020386    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b48679fc2775db635ea16bece4fb656ab5f80fe22b1c9e1f3ae6f9862d0aaf2"
Oct 30 21:54:27.590879 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:27.590838    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:32.999982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:32.999943    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:34.164083 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:34.164050    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:34.492398 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:34.492305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:34.665301 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:34.665270    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"]
Oct 30 21:54:34.668979 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:34.668955    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6317794-llwgp"]
Oct 30 21:54:34.802699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:34.802609    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:35.018212 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:35.018177    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daff3b3a-53b1-4689-8492-c2c045db1b18" path="/var/lib/kubelet/pods/daff3b3a-53b1-4689-8492-c2c045db1b18/volumes"
Oct 30 21:54:35.633331 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:35.633287    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.253200 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:36.253164    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.466312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:36.466273    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.638332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:36.638245    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:38.417751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.417719    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:38.962959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.962924    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"]
Oct 30 21:54:38.963342 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.963324    4410 cpu_manager.go:414] "containerMap: removing container" podUID="a9722350-cb9b-4475-a6cf-447f9d355941" containerName="revoke-keys"
Oct 30 21:54:38.963429 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.963345    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9722350-cb9b-4475-a6cf-447f9d355941" containerName="revoke-keys"
Oct 30 21:54:38.963496 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.963485    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9722350-cb9b-4475-a6cf-447f9d355941" containerName="revoke-keys"
Oct 30 21:54:38.966953 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.966933    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:38.974504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:38.974483    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:39.069046 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.069008    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:39.069228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.069053    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:39.069228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.069186    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:39.069228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.069227    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:54:39.070959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.070939    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"]
Oct 30 21:54:39.098268 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.098247    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.098373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.098281    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.098373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.098324    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546w5\" (UniqueName: \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.144416 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.144382    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:39.199388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.199351    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546w5\" (UniqueName: \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.199566 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.199469    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.199566 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.199517    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.200362 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.200332    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.202054 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.202032    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.258650 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.258591    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546w5\" (UniqueName: \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\") pod \"maas-api-revoke-keys-e2e-ait-282df361-kr7j5\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.279422 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.279398    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:39.649318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:39.649290    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"]
Oct 30 21:54:39.652236 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:54:39.652205    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbbe09a_c516_46ff_b5ac_0e11f8918f8e.slice/crio-73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02 WatchSource:0}: Error finding container 73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02: Status 404 returned error can't find the container with id 73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02
Oct 30 21:54:40.077484 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:40.077394    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5" event={"ID":"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e","Type":"ContainerStarted","Data":"b2d5a9196f248f16a353af33013630c0dca43bf9a8fbe67083eeb93586f719c9"}
Oct 30 21:54:40.077484 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:40.077458    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5" event={"ID":"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e","Type":"ContainerStarted","Data":"73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02"}
Oct 30 21:54:40.190801 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:40.190732    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5" podStartSLOduration=2.190717283 podStartE2EDuration="2.190717283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:40.190294342 +0000 UTC m=+733.279032145" watchObservedRunningTime="2026-10-30 21:54:40.190717283 +0000 UTC m=+733.279455085"
Oct 30 21:54:40.986209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:40.986177    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:41.083681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:41.083651    4410 generic.go:356] "Generic (PLEG): container finished" podID="bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" containerID="b2d5a9196f248f16a353af33013630c0dca43bf9a8fbe67083eeb93586f719c9" exitCode=0
Oct 30 21:54:41.083869 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:41.083710    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5" event={"ID":"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e","Type":"ContainerDied","Data":"b2d5a9196f248f16a353af33013630c0dca43bf9a8fbe67083eeb93586f719c9"}
Oct 30 21:54:41.224395 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:41.224363    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:41.415621 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:41.415584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:42.216119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.216096    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:42.326300 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.326275    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\") pod \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") "
Oct 30 21:54:42.326431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.326313    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\" (UniqueName: \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\") pod \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") "
Oct 30 21:54:42.326431 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.326353    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\") pod \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\" (UID: \"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e\") "
Oct 30 21:54:42.326724 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.326702    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca" pod "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" (UID: "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:42.328290 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.328267    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca" pod "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" (UID: "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:42.328379 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.328323    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5" pod "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" (UID: "bbbbe09a-c516-46ff-b5ac-0e11f8918f8e"). InnerVolumeSpecName "kube-api-access-546w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:42.427521 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.427493    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:42.427521 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.427519    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546w5\" (UniqueName: \"kubernetes.io/projected/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-kube-api-access-546w5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:42.427680 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:42.427530    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:43.095397 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:43.095350    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5" event={"ID":"bbbbe09a-c516-46ff-b5ac-0e11f8918f8e","Type":"ContainerDied","Data":"73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02"}
Oct 30 21:54:43.095397 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:43.095382    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"
Oct 30 21:54:43.095574 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:43.095420    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b3da19a8eb1b3bd7ca9d8a62195027a28d342166af72e8ab468b6e7f984b02"
Oct 30 21:54:43.154382 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:43.154345    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:44.320722 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:44.320691    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:44.496513 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:44.496481    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:44.826180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:44.826141    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:45.093526 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:45.093447    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:45.252526 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:45.252490    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:46.085099 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:46.085065    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:47.448875 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:47.448841    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:48.213997 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:48.213958    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:48.585499 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:48.585415    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:49.826332 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:49.826289    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:50.363334 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:50.363303    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:50.520644 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:54:50.520609    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:54:50.562480 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:50.562449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:51.090738 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:51.090703    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:52.731396 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:52.731362    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:53.001344 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.001259    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:53.170161 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.170132    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:53.273296 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.273216    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"]
Oct 30 21:54:53.321864 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.321835    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f20e6e5e-lpnf9"]
Oct 30 21:54:53.682370 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.682339    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:53.863860 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:53.863821    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:54.029702 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:54.029615    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:55.017885 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.017851    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9722350-cb9b-4475-a6cf-447f9d355941" path="/var/lib/kubelet/pods/a9722350-cb9b-4475-a6cf-447f9d355941/volumes"
Oct 30 21:54:55.226267 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.226234    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:55.429986 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.429955    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:55.588203 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.588159    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:55.769617 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.769547    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"]
Oct 30 21:54:55.818789 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:55.818749    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-282df361-kr7j5"]
Oct 30 21:54:56.339743 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:56.339709    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:56.889127 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:56.889082    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:57.018047 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:57.018016    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" path="/var/lib/kubelet/pods/bbbbe09a-c516-46ff-b5ac-0e11f8918f8e/volumes"
Oct 30 21:54:57.855425 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:57.855394    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.109664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.109588    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.275785 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.275739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.454722 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.454662    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.646409 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.646374    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.807873 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.807791    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:58.969241 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:58.969202    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:59.141414 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:54:59.141377    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:03.226706 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:03.226669    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:03.392278 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:03.392248    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:03.561661 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:03.561575    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:03.896028 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:03.895989    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:07.324269 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:07.324232    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:07.534251 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:07.534216    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:08.833302 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:08.833261    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:09.167589 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:09.167557    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:10.904635 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:10.904555    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.087665 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:11.087636    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.317247 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:11.317169    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.543119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:11.543076    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:11.698234 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:11.698180    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:14.187478 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:14.187444    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:17.541390 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:17.541356    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:18.282389 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:18.282352    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:18.474352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:18.474320    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:19.735580 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:19.735528    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:20.521798 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:55:20.521750    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:55:24.613357 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:24.613319    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:24.813994 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:24.813963    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:24.970305 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:24.970276    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:26.844670 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:26.844627    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:28.173388 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:28.173354    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:28.387356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:28.387322    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:28.559945 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:28.559864    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:29.467360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:29.467328    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:31.185541 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.185504    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:31.351805 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.351752    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:31.597642 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.597552    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:31.798708 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.798659    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:31.952529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.952496    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 21:55:31.952856 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.952844    4410 cpu_manager.go:414] "containerMap: removing container" podUID="bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" containerName="revoke-keys"
Oct 30 21:55:31.952908 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.952857    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" containerName="revoke-keys"
Oct 30 21:55:31.952943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.952927    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbbe09a-c516-46ff-b5ac-0e11f8918f8e" containerName="revoke-keys"
Oct 30 21:55:31.956139 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.956119    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:31.991504 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:31.991466    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:32.008528 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.008505    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"openshift-service-ca.crt\""
Oct 30 21:55:32.051169 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051140    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.051316 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051183    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.051316 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051288    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.051423 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051326    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws89v\" (UniqueName: \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.051485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051450    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.051538 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.051485    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.056996 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.056976    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"kube-root-ca.crt\""
Oct 30 21:55:32.058554 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.058537    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 21:55:32.063440 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.063425    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"simulator-w3-e90e35-kserve-self-signed-certs\""
Oct 30 21:55:32.099373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.099356    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"default-dockercfg-rxsvw\""
Oct 30 21:55:32.152828 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.152800    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.152840    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws89v\" (UniqueName: \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.152886    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.152905    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.152929    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153221 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.153064    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153369 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.153345    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153449 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.153366    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.153449 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.153418    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.155092 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.155073    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.155338 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.155321    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.181459 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.181414    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:32.209351 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.209298    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws89v\" (UniqueName: \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\") pod \"simulator-w3-e90e35-kserve-696f7c57d5-jgl6z\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") " pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.267872 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.267848    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:32.638787 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.638584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:32.638787 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:32.638713    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 21:55:32.640924 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:55:32.640899    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc62a0376_d5cf_4f17_95b1_f0c6ba5c6982.slice/crio-50f2a0ae83cdb3e254d1fe9e6b9450c6c0729b7d84e1fbbf974b017557722a0e WatchSource:0}: Error finding container 50f2a0ae83cdb3e254d1fe9e6b9450c6c0729b7d84e1fbbf974b017557722a0e: Status 404 returned error can't find the container with id 50f2a0ae83cdb3e254d1fe9e6b9450c6c0729b7d84e1fbbf974b017557722a0e
Oct 30 21:55:33.320052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:33.320017    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" event={"ID":"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982","Type":"ContainerStarted","Data":"50f2a0ae83cdb3e254d1fe9e6b9450c6c0729b7d84e1fbbf974b017557722a0e"}
Oct 30 21:55:33.382383 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:33.382341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:33.655820 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:33.655769    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:35.331587 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:35.331500    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" event={"ID":"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982","Type":"ContainerStarted","Data":"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"}
Oct 30 21:55:35.450686 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:35.450630    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podStartSLOduration=2.027096744 podStartE2EDuration="4.450611207s" totalImagesPullingTime="2.423514463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:35.449345963 +0000 UTC m=+788.538083766" watchObservedRunningTime="2026-10-30 21:55:35.450611207 +0000 UTC m=+788.539349011"
Oct 30 21:55:36.268954 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:36.268923    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:37.076874 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:37.076840    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:37.518377 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:37.518349    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:37.705940 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:37.705906    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:37.897992 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:37.897963    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:40.750547 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:40.750516    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:40.943855 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:40.943822    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.235716 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:42.235677    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.268138 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:42.268110    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:42.281149 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:42.281123    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:42.376142 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:42.376117    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 21:55:44.922477 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:44.922445    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.551646 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:45.551613    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.730723 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:45.730686    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:47.744535 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:47.744502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:50.521231 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:55:50.521194    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:55:52.576640 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:52.576607    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:57.736719 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:57.736674    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:58.693882 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:58.693843    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:59.289959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.289929    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 21:55:59.293069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.293047    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385493    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385577    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385684    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385887 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385733    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385887 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385842    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.385887 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.385879    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ms6\" (UniqueName: \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.401201 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.401177    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"openshift-service-ca.crt\""
Oct 30 21:55:59.401442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.401424    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"kube-root-ca.crt\""
Oct 30 21:55:59.401884 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.401863    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"default-dockercfg-6s4ht\""
Oct 30 21:55:59.401884 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.401875    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"simulator-w0-b8717e-kserve-self-signed-certs\""
Oct 30 21:55:59.405988 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.405965    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 21:55:59.419637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.419609    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:59.486434 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486403    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486449    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486482    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486542    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486592    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486623    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ms6\" (UniqueName: \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.486962 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.486893    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.487147 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.487126    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.487266 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.487245    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.489089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.489068    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.489177 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.489164    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.546250 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.546185    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ms6\" (UniqueName: \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\") pod \"simulator-w0-b8717e-kserve-7c4bfff749-rzdxd\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") " pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.604739 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.604706    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:55:59.693681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.693650    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:59.981982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:55:59.981874    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 21:55:59.984196 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:55:59.984169    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcac9b08b_7aa9_401c_87e4_8be2eef06427.slice/crio-ca10961dd95c87e63ba94bd3a0b04d027f3a83086797bfad1652f77ae7d31401 WatchSource:0}: Error finding container ca10961dd95c87e63ba94bd3a0b04d027f3a83086797bfad1652f77ae7d31401: Status 404 returned error can't find the container with id ca10961dd95c87e63ba94bd3a0b04d027f3a83086797bfad1652f77ae7d31401
Oct 30 21:56:00.445310 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:00.445276    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" event={"ID":"cac9b08b-7aa9-401c-87e4-8be2eef06427","Type":"ContainerStarted","Data":"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"}
Oct 30 21:56:00.445674 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:00.445320    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" event={"ID":"cac9b08b-7aa9-401c-87e4-8be2eef06427","Type":"ContainerStarted","Data":"ca10961dd95c87e63ba94bd3a0b04d027f3a83086797bfad1652f77ae7d31401"}
Oct 30 21:56:00.564695 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:00.564638    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podStartSLOduration=1.5646245109999999 podStartE2EDuration="1.564624511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:00.563906413 +0000 UTC m=+813.652644216" watchObservedRunningTime="2026-10-30 21:56:00.564624511 +0000 UTC m=+813.653362344"
Oct 30 21:56:00.605237 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:00.605204    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:56:03.446899 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:03.446867    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:03.653522 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:03.653460    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:03.814036 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:03.813941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:06.578345 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:06.578305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:07.340705 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:07.340639    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:07.614727 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:07.614649    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:09.057365 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:09.057334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:09.605275 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:09.605234    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:56:09.617658 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:09.617636    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:56:10.498138 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:10.498107    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 21:56:13.991614 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:13.991576    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:17.903228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:17.903189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:18.231419 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.231343    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 21:56:18.234712 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.234695    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.337709 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.337672    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"premium-w3-e90e35-kserve-self-signed-certs\""
Oct 30 21:56:18.338304 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.338275    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:18.339438 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.339416    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 21:56:18.358203 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358177    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.358323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358225    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqkn\" (UniqueName: \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.358323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358295    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.358323 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358311    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.358484 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358326    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.358484 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.358361    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459348 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459312    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459364    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqkn\" (UniqueName: \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459416    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459437    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459459    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.459529 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459496    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.460035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.459998    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.460035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.460013    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.460218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.460041    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.461924 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.461902    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.462074 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.462057    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.517128 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.517063    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqkn\" (UniqueName: \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\") pod \"premium-w3-e90e35-kserve-748b8d7548-hlnhc\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") " pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.546590 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.546560    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:18.926226 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.926202    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 21:56:18.928572 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:56:18.928545    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5444c7_6aa1_41f2_8604_4a6864fd7f55.slice/crio-2755f725c3fae7ae14019b4051d68fbcc3969a52719bff5d9a5247b4d8c76bc9 WatchSource:0}: Error finding container 2755f725c3fae7ae14019b4051d68fbcc3969a52719bff5d9a5247b4d8c76bc9: Status 404 returned error can't find the container with id 2755f725c3fae7ae14019b4051d68fbcc3969a52719bff5d9a5247b4d8c76bc9
Oct 30 21:56:18.971268 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:18.971241    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:19.529139 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:19.529104    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" event={"ID":"fb5444c7-6aa1-41f2-8604-4a6864fd7f55","Type":"ContainerStarted","Data":"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"}
Oct 30 21:56:19.529292 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:19.529146    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" event={"ID":"fb5444c7-6aa1-41f2-8604-4a6864fd7f55","Type":"ContainerStarted","Data":"2755f725c3fae7ae14019b4051d68fbcc3969a52719bff5d9a5247b4d8c76bc9"}
Oct 30 21:56:19.547540 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:19.547509    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:19.646925 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:19.646874    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podStartSLOduration=1.646859449 podStartE2EDuration="1.646859449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:19.644934157 +0000 UTC m=+832.733671960" watchObservedRunningTime="2026-10-30 21:56:19.646859449 +0000 UTC m=+832.735597251"
Oct 30 21:56:20.522181 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:56:20.522148    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:56:23.863661 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:23.863629    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.051209 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:28.051174    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.547477 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:28.547442    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:28.559997 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:28.559968    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:28.586034 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:28.586011    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 21:56:28.888066 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:28.887978    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:30.026987 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.026952    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 21:56:30.030417 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.030399    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.133133 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.133102    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"kube-root-ca.crt\""
Oct 30 21:56:30.133392 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.133107    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"openshift-service-ca.crt\""
Oct 30 21:56:30.133465 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.133152    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"simulator-w4-d2a3e0-kserve-self-signed-certs\""
Oct 30 21:56:30.133636 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.133617    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"default-dockercfg-7kbkr\""
Oct 30 21:56:30.136474 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.136454    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 21:56:30.168963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.168936    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.169163 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.168985    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxshd\" (UniqueName: \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.169163 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.169027    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.169163 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.169049    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.169288 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.169179    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.169288 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.169206    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.251982 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.251929    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:30.270052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270018    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270052 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270055    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270092    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270125    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxshd\" (UniqueName: \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270163    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270191    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270461 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270437    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270528 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270492    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.270659 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.270639    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.272416 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.272395    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.272714 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.272694    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.329134 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.329060    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxshd\" (UniqueName: \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\") pod \"simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") " pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.342942 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.342915    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:30.734990 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:30.734966    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 21:56:30.737561 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:56:30.737532    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36673df_c797_4dba_866d_8351c430c91f.slice/crio-fab8da2af6aa7318ad07771a6bcd92a553969ed1e0ce31bdc35d063c0653ad28 WatchSource:0}: Error finding container fab8da2af6aa7318ad07771a6bcd92a553969ed1e0ce31bdc35d063c0653ad28: Status 404 returned error can't find the container with id fab8da2af6aa7318ad07771a6bcd92a553969ed1e0ce31bdc35d063c0653ad28
Oct 30 21:56:31.591539 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:31.591501    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" event={"ID":"e36673df-c797-4dba-866d-8351c430c91f","Type":"ContainerStarted","Data":"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"}
Oct 30 21:56:31.591955 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:31.591552    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" event={"ID":"e36673df-c797-4dba-866d-8351c430c91f","Type":"ContainerStarted","Data":"fab8da2af6aa7318ad07771a6bcd92a553969ed1e0ce31bdc35d063c0653ad28"}
Oct 30 21:56:31.715755 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:31.715698    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podStartSLOduration=2.715683841 podStartE2EDuration="2.715683841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:31.714404393 +0000 UTC m=+844.803142206" watchObservedRunningTime="2026-10-30 21:56:31.715683841 +0000 UTC m=+844.804421647"
Oct 30 21:56:32.343572 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:32.343541    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:33.953556 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:33.953513    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:38.840484 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:38.840448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:40.343084 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.343001    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:40.355405 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.355379    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:40.649285 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.649259    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 21:56:40.853475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.853443    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 21:56:40.857307 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.857287    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.911129 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.911025    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"openshift-service-ca.crt\""
Oct 30 21:56:40.911257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.911201    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"default-dockercfg-ft4lg\""
Oct 30 21:56:40.911257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.911200    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"kube-root-ca.crt\""
Oct 30 21:56:40.911257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.911237    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"simulator-w2-90e0d5-kserve-self-signed-certs\""
Oct 30 21:56:40.917368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.917341    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 21:56:40.958291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958264    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.958291 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958294    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.958441 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958309    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6d2\" (UniqueName: \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.958497 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958481    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.958531 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958509    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.958567 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.958539    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:40.965192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:40.965168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:41.060107 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060071    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060122    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060166    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060214    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060406 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060335    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060406 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060369    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6d2\" (UniqueName: \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060544 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060517    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.060622 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060584    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.061003 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.060977    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.062880 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.062859    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.062974 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.062889    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.117951 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.117921    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6d2\" (UniqueName: \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\") pod \"simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") " pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.169220 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.169147    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:41.546198 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.546139    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 21:56:41.548682 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:56:41.548656    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod111d0a19_755a_4ba5_ad08_ae266fa61f76.slice/crio-25daeffd08741146af21e6eabf1d13d4f8dbbd5835d600f71067bd262716df65 WatchSource:0}: Error finding container 25daeffd08741146af21e6eabf1d13d4f8dbbd5835d600f71067bd262716df65: Status 404 returned error can't find the container with id 25daeffd08741146af21e6eabf1d13d4f8dbbd5835d600f71067bd262716df65
Oct 30 21:56:41.644963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.644930    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" event={"ID":"111d0a19-755a-4ba5-ad08-ae266fa61f76","Type":"ContainerStarted","Data":"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"}
Oct 30 21:56:41.645103 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.644976    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" event={"ID":"111d0a19-755a-4ba5-ad08-ae266fa61f76","Type":"ContainerStarted","Data":"25daeffd08741146af21e6eabf1d13d4f8dbbd5835d600f71067bd262716df65"}
Oct 30 21:56:41.760352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:41.760305    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podStartSLOduration=1.760288277 podStartE2EDuration="1.760288277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:41.759766039 +0000 UTC m=+854.848503853" watchObservedRunningTime="2026-10-30 21:56:41.760288277 +0000 UTC m=+854.849026079"
Oct 30 21:56:42.169759 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:42.169719    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:43.280943 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:43.280907    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:44.071979 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:44.071943    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:45.435389 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:45.435350    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:49.173994 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:49.173955    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:50.521353 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:56:50.521321    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:56:51.169560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:51.169528    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:51.181992 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:51.181970    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:51.706731 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:51.706703    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 21:56:52.841677 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:52.841641    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:54.090191 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:54.090155    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:58.938915 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:56:58.938877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:03.821162 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:03.821128    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:04.039544 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:04.039507    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:08.768468 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:08.768430    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:13.645735 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:13.645689    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:15.567817 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.567766    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 21:57:15.570902 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.570884    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.672664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.672634    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"premium-w4-d2a3e0-kserve-self-signed-certs\""
Oct 30 21:57:15.674498 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.674469    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 21:57:15.674932 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.674913    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.675018 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.674972    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.675119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.675096    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.675155 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.675138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452pj\" (UniqueName: \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.675223 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.675205    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.675266 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.675252    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.690225 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.690194    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:15.776190 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776161    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776216    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776239    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452pj\" (UniqueName: \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776260    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776289    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776368 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776339    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776636 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776608    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776751 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776676    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.776814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.776754    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.778597 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.778577    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.778863 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.778842    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.833888 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.833824    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-452pj\" (UniqueName: \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\") pod \"premium-w4-d2a3e0-kserve-7d76b67b47-9gq48\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:15.884736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:15.884699    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:16.265932 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:16.265902    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 21:57:16.268403 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:57:16.268371    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eef84fe_4003_47f8_b058_f3f6e58369f0.slice/crio-e2393eab6adcfd53b1307bf737401084871ac547e4e96355fb57343540420019 WatchSource:0}: Error finding container e2393eab6adcfd53b1307bf737401084871ac547e4e96355fb57343540420019: Status 404 returned error can't find the container with id e2393eab6adcfd53b1307bf737401084871ac547e4e96355fb57343540420019
Oct 30 21:57:16.828257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:16.828217    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" event={"ID":"4eef84fe-4003-47f8-b058-f3f6e58369f0","Type":"ContainerStarted","Data":"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"}
Oct 30 21:57:16.828722 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:16.828268    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" event={"ID":"4eef84fe-4003-47f8-b058-f3f6e58369f0","Type":"ContainerStarted","Data":"e2393eab6adcfd53b1307bf737401084871ac547e4e96355fb57343540420019"}
Oct 30 21:57:16.885487 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:16.885449    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:16.941768 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:16.941690    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podStartSLOduration=1.9416729419999998 podStartE2EDuration="1.941672942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:16.940873758 +0000 UTC m=+890.029611573" watchObservedRunningTime="2026-10-30 21:57:16.941672942 +0000 UTC m=+890.030410746"
Oct 30 21:57:18.238669 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:18.238635    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:18.555059 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:18.554971    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:20.521347 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:57:20.521310    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:57:21.613218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:21.613185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:21.786883 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:21.786845    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:23.375951 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:23.375920    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:25.885676 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:25.885643    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:25.898201 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:25.898180    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:26.886840 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:26.886812    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 21:57:28.172679 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:28.172637    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:32.976427 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:32.976390    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:37.840469 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:37.840414    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:40.719460 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:40.719425    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:42.663132 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:42.663092    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:46.932137 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:46.932101    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"]
Oct 30 21:57:46.935673 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:46.935655    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:46.988060 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:46.988031    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:47.037312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.037284    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:57:47.037438 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.037321    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:47.037438 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.037344    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:47.039478 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.039459    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"]
Oct 30 21:57:47.075860 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.075837    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.075989 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.075972    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.076035 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.075996    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpxbb\" (UniqueName: \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.177413 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.177374    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.177608 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.177508    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpxbb\" (UniqueName: \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.177681 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.177605    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.178109 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.178086    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.179850 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.179834    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.234112 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.234058    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpxbb\" (UniqueName: \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\") pod \"maas-api-revoke-sub-52aefa85-87dps\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.248258 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.248240    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:47.588676 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.588593    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:47.620227 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.620194    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"]
Oct 30 21:57:47.622385 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:57:47.622355    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee06c6bb_9155_4737_8224_b2ce32c91416.slice/crio-fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046 WatchSource:0}: Error finding container fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046: Status 404 returned error can't find the container with id fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046
Oct 30 21:57:47.976145 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.976111    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps" event={"ID":"ee06c6bb-9155-4737-8224-b2ce32c91416","Type":"ContainerStarted","Data":"4d96b514059ab3a94b53d1e8b9d02326247b60298afa41664efa0978c1a79dde"}
Oct 30 21:57:47.976520 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:47.976165    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps" event={"ID":"ee06c6bb-9155-4737-8224-b2ce32c91416","Type":"ContainerStarted","Data":"fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046"}
Oct 30 21:57:48.095903 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:48.095831    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps" podStartSLOduration=2.095813412 podStartE2EDuration="2.095813412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:48.093460633 +0000 UTC m=+921.182198436" watchObservedRunningTime="2026-10-30 21:57:48.095813412 +0000 UTC m=+921.184551212"
Oct 30 21:57:48.982669 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:48.982633    4410 generic.go:356] "Generic (PLEG): container finished" podID="ee06c6bb-9155-4737-8224-b2ce32c91416" containerID="4d96b514059ab3a94b53d1e8b9d02326247b60298afa41664efa0978c1a79dde" exitCode=0
Oct 30 21:57:48.983087 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:48.982721    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps" event={"ID":"ee06c6bb-9155-4737-8224-b2ce32c91416","Type":"ContainerDied","Data":"4d96b514059ab3a94b53d1e8b9d02326247b60298afa41664efa0978c1a79dde"}
Oct 30 21:57:49.218373 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:49.218331    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:50.113272 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.113247    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:50.206071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.206044    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\") pod \"ee06c6bb-9155-4737-8224-b2ce32c91416\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") "
Oct 30 21:57:50.206213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.206144    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\") pod \"ee06c6bb-9155-4737-8224-b2ce32c91416\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") "
Oct 30 21:57:50.206213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.206192    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\" (UniqueName: \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\") pod \"ee06c6bb-9155-4737-8224-b2ce32c91416\" (UID: \"ee06c6bb-9155-4737-8224-b2ce32c91416\") "
Oct 30 21:57:50.206453 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.206427    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca" pod "ee06c6bb-9155-4737-8224-b2ce32c91416" (UID: "ee06c6bb-9155-4737-8224-b2ce32c91416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:50.208172 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.208148    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb" pod "ee06c6bb-9155-4737-8224-b2ce32c91416" (UID: "ee06c6bb-9155-4737-8224-b2ce32c91416"). InnerVolumeSpecName "kube-api-access-gpxbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:50.208288 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.208225    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca" pod "ee06c6bb-9155-4737-8224-b2ce32c91416" (UID: "ee06c6bb-9155-4737-8224-b2ce32c91416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:50.307622 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.307556    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:50.307622 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.307584    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee06c6bb-9155-4737-8224-b2ce32c91416-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:50.307622 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.307599    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpxbb\" (UniqueName: \"kubernetes.io/projected/ee06c6bb-9155-4737-8224-b2ce32c91416-kube-api-access-gpxbb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:50.521322 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:57:50.521291    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:57:50.993276 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.993250    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"
Oct 30 21:57:50.993447 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.993280    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps" event={"ID":"ee06c6bb-9155-4737-8224-b2ce32c91416","Type":"ContainerDied","Data":"fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046"}
Oct 30 21:57:50.993447 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:50.993343    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd1722b775a5cfd2c602ad096744ac279418a46c7b62d3f9c49bafd786a72046"
Oct 30 21:57:52.372224 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:52.372185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:57.195112 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:57.195049    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:59.937543 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:57:59.937506    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:00.086865 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:00.086828    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:01.946177 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:01.946140    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:04.858900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.858866    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"]
Oct 30 21:58:04.859257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.859239    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ee06c6bb-9155-4737-8224-b2ce32c91416" containerName="revoke-keys"
Oct 30 21:58:04.859257 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.859250    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee06c6bb-9155-4737-8224-b2ce32c91416" containerName="revoke-keys"
Oct 30 21:58:04.859343 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.859330    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee06c6bb-9155-4737-8224-b2ce32c91416" containerName="revoke-keys"
Oct 30 21:58:04.862532 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.862515    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:04.914790 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.914733    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:58:04.914790 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.914752    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:04.934809 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.934765    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:04.934964 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.934945    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5nt5\" (UniqueName: \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:04.935015 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.934980    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:04.964152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.964125    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:04.964270 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.964174    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:04.966015 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:04.965996    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"]
Oct 30 21:58:05.036180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.036158    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5nt5\" (UniqueName: \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.036309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.036192    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.036309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.036249    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.037012 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.036990    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.038550 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.038529    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.092666 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.092637    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5nt5\" (UniqueName: \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\") pod \"maas-api-revoke-sub-22858176-stzjw\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.174882 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.174858    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:05.545160 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:05.545134    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"]
Oct 30 21:58:05.547189 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:05.547150    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18069dd1_417e_4eb6_a1a0_570913d37ce8.slice/crio-b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017 WatchSource:0}: Error finding container b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017: Status 404 returned error can't find the container with id b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017
Oct 30 21:58:06.065409 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:06.065373    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw" event={"ID":"18069dd1-417e-4eb6-a1a0-570913d37ce8","Type":"ContainerStarted","Data":"175f385514f448fbb605744370294d4b323ac05e285904b47ccb646cc2c3db10"}
Oct 30 21:58:06.065855 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:06.065452    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw" event={"ID":"18069dd1-417e-4eb6-a1a0-570913d37ce8","Type":"ContainerStarted","Data":"b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017"}
Oct 30 21:58:06.191947 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:06.191899    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw" podStartSLOduration=2.191886683 podStartE2EDuration="2.191886683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:06.190767262 +0000 UTC m=+939.279505066" watchObservedRunningTime="2026-10-30 21:58:06.191886683 +0000 UTC m=+939.280624485"
Oct 30 21:58:06.834135 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:06.834098    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:07.071199 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:07.071161    4410 generic.go:356] "Generic (PLEG): container finished" podID="18069dd1-417e-4eb6-a1a0-570913d37ce8" containerID="175f385514f448fbb605744370294d4b323ac05e285904b47ccb646cc2c3db10" exitCode=0
Oct 30 21:58:07.071570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:07.071217    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw" event={"ID":"18069dd1-417e-4eb6-a1a0-570913d37ce8","Type":"ContainerDied","Data":"175f385514f448fbb605744370294d4b323ac05e285904b47ccb646cc2c3db10"}
Oct 30 21:58:08.201985 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.201959    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:08.268345 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.268306    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\") pod \"18069dd1-417e-4eb6-a1a0-570913d37ce8\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") "
Oct 30 21:58:08.268485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.268390    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\" (UniqueName: \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\") pod \"18069dd1-417e-4eb6-a1a0-570913d37ce8\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") "
Oct 30 21:58:08.268485 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.268450    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\") pod \"18069dd1-417e-4eb6-a1a0-570913d37ce8\" (UID: \"18069dd1-417e-4eb6-a1a0-570913d37ce8\") "
Oct 30 21:58:08.268833 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.268804    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca" pod "18069dd1-417e-4eb6-a1a0-570913d37ce8" (UID: "18069dd1-417e-4eb6-a1a0-570913d37ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:08.270432 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.270412    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca" pod "18069dd1-417e-4eb6-a1a0-570913d37ce8" (UID: "18069dd1-417e-4eb6-a1a0-570913d37ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:08.270511 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.270468    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5" pod "18069dd1-417e-4eb6-a1a0-570913d37ce8" (UID: "18069dd1-417e-4eb6-a1a0-570913d37ce8"). InnerVolumeSpecName "kube-api-access-v5nt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:08.369637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.369569    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:08.369637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.369592    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5nt5\" (UniqueName: \"kubernetes.io/projected/18069dd1-417e-4eb6-a1a0-570913d37ce8-kube-api-access-v5nt5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:08.369637 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:08.369601    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18069dd1-417e-4eb6-a1a0-570913d37ce8-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:09.081160 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:09.081133    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"
Oct 30 21:58:09.081393 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:09.081170    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw" event={"ID":"18069dd1-417e-4eb6-a1a0-570913d37ce8","Type":"ContainerDied","Data":"b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017"}
Oct 30 21:58:09.081393 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:09.081214    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3f85b9b1bb41287e78b477e87bf5e97395901d7f530e5852f86b6c9972a2017"
Oct 30 21:58:11.651089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:11.650998    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:12.388182 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.388151    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 21:58:12.388538 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.388526    4410 cpu_manager.go:414] "containerMap: removing container" podUID="18069dd1-417e-4eb6-a1a0-570913d37ce8" containerName="revoke-keys"
Oct 30 21:58:12.388593 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.388539    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18069dd1-417e-4eb6-a1a0-570913d37ce8" containerName="revoke-keys"
Oct 30 21:58:12.388631 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.388597    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="18069dd1-417e-4eb6-a1a0-570913d37ce8" containerName="revoke-keys"
Oct 30 21:58:12.391944 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.391924    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.492685 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.492651    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"premium-w1-1c3eef-kserve-self-signed-certs\""
Oct 30 21:58:12.492988 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.492970    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"default-dockercfg-dc4hx\""
Oct 30 21:58:12.493161 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.493144    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"kube-root-ca.crt\""
Oct 30 21:58:12.493220 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.493199    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"openshift-service-ca.crt\""
Oct 30 21:58:12.495053 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.495033    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 21:58:12.505234 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505207    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.505386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505289    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.505386 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505334    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.505489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505418    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rvl\" (UniqueName: \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.505489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505455    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.505581 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.505565    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.518451 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.518420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:12.606632 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606597    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.606852 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606669    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.606852 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606696    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.606852 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606735    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.606852 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606800    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rvl\" (UniqueName: \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.606852 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.606838    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.607147 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.607129    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.607208 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.607160    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.607271 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.607222    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.609604 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.609573    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.609754 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.609737    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.663558 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.663533    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rvl\" (UniqueName: \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\") pod \"premium-w1-1c3eef-kserve-d586cf5f6-fn2dd\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") " pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:12.703496 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:12.703466    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:13.083987 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:13.083963    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 21:58:13.086449 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:13.086418    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f734874_6279_4e14_962d_10fe1c71ac16.slice/crio-16cfcbbd9db61dce8a82b74aa2112e8243bc94ad40bbb5d010919d38b582d533 WatchSource:0}: Error finding container 16cfcbbd9db61dce8a82b74aa2112e8243bc94ad40bbb5d010919d38b582d533: Status 404 returned error can't find the container with id 16cfcbbd9db61dce8a82b74aa2112e8243bc94ad40bbb5d010919d38b582d533
Oct 30 21:58:13.102735 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:13.102709    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" event={"ID":"8f734874-6279-4e14-962d-10fe1c71ac16","Type":"ContainerStarted","Data":"16cfcbbd9db61dce8a82b74aa2112e8243bc94ad40bbb5d010919d38b582d533"}
Oct 30 21:58:14.108664 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:14.108629    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" event={"ID":"8f734874-6279-4e14-962d-10fe1c71ac16","Type":"ContainerStarted","Data":"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"}
Oct 30 21:58:14.223123 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:14.223066    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podStartSLOduration=2.223047793 podStartE2EDuration="2.223047793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.221867088 +0000 UTC m=+947.310604891" watchObservedRunningTime="2026-10-30 21:58:14.223047793 +0000 UTC m=+947.311785664"
Oct 30 21:58:14.704084 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:14.704037    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:16.590535 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:16.590502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:18.563356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:18.563325    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:20.521675 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:58:20.521638    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:58:21.480900 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:21.480863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.373125 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.373094    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"]
Oct 30 21:58:22.376701 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.376675    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.478075 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.478048    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:22.478228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.478098    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:22.478228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.478105    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:22.478228 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.478157    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:58:22.480092 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.480075    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"]
Oct 30 21:58:22.499541 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.499515    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.499659 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.499620    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866vq\" (UniqueName: \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.499659 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.499656    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.601028 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.601004    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866vq\" (UniqueName: \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.601119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.601037    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.601119 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.601084    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.601746 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.601725    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.603342 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.603324    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.658631 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.658604    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866vq\" (UniqueName: \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\") pod \"maas-api-revoke-sub-932d0ecb-zclsg\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.689458 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.689437    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:22.704477 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.704451    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:22.720144 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.720118    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:22.915089 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.915012    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"]
Oct 30 21:58:22.918744 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.918724    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:22.975299 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:22.975269    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"]
Oct 30 21:58:23.006583 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.006553    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.006721 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.006618    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.006721 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.006675    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86q9h\" (UniqueName: \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.062957 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.062934    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"]
Oct 30 21:58:23.064544 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:23.064522    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947838c7_f014_4695_b9b8_04e78bfc5a21.slice/crio-18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738 WatchSource:0}: Error finding container 18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738: Status 404 returned error can't find the container with id 18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738
Oct 30 21:58:23.107616 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.107593    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.107717 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.107646    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.107717 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.107664    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86q9h\" (UniqueName: \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.108509 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.108491    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.109977 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.109957    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.151442 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.151420    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg" event={"ID":"947838c7-f014-4695-b9b8-04e78bfc5a21","Type":"ContainerStarted","Data":"18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738"}
Oct 30 21:58:23.162607 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.162578    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 21:58:23.164044 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.164027    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86q9h\" (UniqueName: \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\") pod \"maas-api-revoke-sub-e1226f44-rvt4d\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.231470 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.231418    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:23.612591 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:23.612567    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"]
Oct 30 21:58:23.614701 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:23.614676    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45940d30_4d56_4c45_90be_14c4b5006935.slice/crio-67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a WatchSource:0}: Error finding container 67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a: Status 404 returned error can't find the container with id 67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a
Oct 30 21:58:24.158330 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:24.158298    4410 generic.go:356] "Generic (PLEG): container finished" podID="947838c7-f014-4695-b9b8-04e78bfc5a21" containerID="3db9b28d30108c65261f99dee0888d24b5fcb9468c97eaef8eaf48d0ec06d593" exitCode=0
Oct 30 21:58:24.158489 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:24.158391    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg" event={"ID":"947838c7-f014-4695-b9b8-04e78bfc5a21","Type":"ContainerDied","Data":"3db9b28d30108c65261f99dee0888d24b5fcb9468c97eaef8eaf48d0ec06d593"}
Oct 30 21:58:24.159904 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:24.159878    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d" event={"ID":"45940d30-4d56-4c45-90be-14c4b5006935","Type":"ContainerStarted","Data":"3e7d50414ad33675c879994b991f8e0db54e57695dacbffb887af54c36770c84"}
Oct 30 21:58:24.160011 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:24.159925    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d" event={"ID":"45940d30-4d56-4c45-90be-14c4b5006935","Type":"ContainerStarted","Data":"67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a"}
Oct 30 21:58:25.166595 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.166564    4410 generic.go:356] "Generic (PLEG): container finished" podID="45940d30-4d56-4c45-90be-14c4b5006935" containerID="3e7d50414ad33675c879994b991f8e0db54e57695dacbffb887af54c36770c84" exitCode=0
Oct 30 21:58:25.167038 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.166646    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d" event={"ID":"45940d30-4d56-4c45-90be-14c4b5006935","Type":"ContainerDied","Data":"3e7d50414ad33675c879994b991f8e0db54e57695dacbffb887af54c36770c84"}
Oct 30 21:58:25.309986 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.309962    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:25.432159 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.432091    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\") pod \"947838c7-f014-4695-b9b8-04e78bfc5a21\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") "
Oct 30 21:58:25.432278 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.432187    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\") pod \"947838c7-f014-4695-b9b8-04e78bfc5a21\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") "
Oct 30 21:58:25.432278 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.432252    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\" (UniqueName: \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\") pod \"947838c7-f014-4695-b9b8-04e78bfc5a21\" (UID: \"947838c7-f014-4695-b9b8-04e78bfc5a21\") "
Oct 30 21:58:25.432568 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.432544    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca" pod "947838c7-f014-4695-b9b8-04e78bfc5a21" (UID: "947838c7-f014-4695-b9b8-04e78bfc5a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:25.434140 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.434109    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq" pod "947838c7-f014-4695-b9b8-04e78bfc5a21" (UID: "947838c7-f014-4695-b9b8-04e78bfc5a21"). InnerVolumeSpecName "kube-api-access-866vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:25.434284 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.434267    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca" pod "947838c7-f014-4695-b9b8-04e78bfc5a21" (UID: "947838c7-f014-4695-b9b8-04e78bfc5a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:25.533124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.533088    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:25.533124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.533119    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947838c7-f014-4695-b9b8-04e78bfc5a21-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:25.533307 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:25.533136    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866vq\" (UniqueName: \"kubernetes.io/projected/947838c7-f014-4695-b9b8-04e78bfc5a21-kube-api-access-866vq\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:26.173210 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.173179    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"
Oct 30 21:58:26.173595 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.173215    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg" event={"ID":"947838c7-f014-4695-b9b8-04e78bfc5a21","Type":"ContainerDied","Data":"18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738"}
Oct 30 21:58:26.173595 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.173262    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18c98ff297cb37d3cab33d5d8bed24de3f9016a4d2bda11a735d333db4b98738"
Oct 30 21:58:26.314753 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.314726    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:26.366490 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.366452    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:26.443360 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.443276    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\") pod \"45940d30-4d56-4c45-90be-14c4b5006935\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") "
Oct 30 21:58:26.443519 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.443395    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\" (UniqueName: \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\") pod \"45940d30-4d56-4c45-90be-14c4b5006935\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") "
Oct 30 21:58:26.443519 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.443493    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\") pod \"45940d30-4d56-4c45-90be-14c4b5006935\" (UID: \"45940d30-4d56-4c45-90be-14c4b5006935\") "
Oct 30 21:58:26.443662 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.443638    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca" pod "45940d30-4d56-4c45-90be-14c4b5006935" (UID: "45940d30-4d56-4c45-90be-14c4b5006935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:26.443835 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.443817    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:26.445553 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.445526    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca" pod "45940d30-4d56-4c45-90be-14c4b5006935" (UID: "45940d30-4d56-4c45-90be-14c4b5006935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:26.445553 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.445539    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h" pod "45940d30-4d56-4c45-90be-14c4b5006935" (UID: "45940d30-4d56-4c45-90be-14c4b5006935"). InnerVolumeSpecName "kube-api-access-86q9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:26.545156 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.545130    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86q9h\" (UniqueName: \"kubernetes.io/projected/45940d30-4d56-4c45-90be-14c4b5006935-kube-api-access-86q9h\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:26.545156 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:26.545155    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45940d30-4d56-4c45-90be-14c4b5006935-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:27.178814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:27.178765    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"
Oct 30 21:58:27.178814 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:27.178799    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d" event={"ID":"45940d30-4d56-4c45-90be-14c4b5006935","Type":"ContainerDied","Data":"67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a"}
Oct 30 21:58:27.179302 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:27.178848    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67375705dde20e50523df5756a4e8298d2d759b2580b515b5240ad312dd7e70a"
Oct 30 21:58:31.163105 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:31.163063    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:35.963494 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:35.963458    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:40.760640 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:40.760598    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:42.965907 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.965861    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"]
Oct 30 21:58:42.966277 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966269    4410 cpu_manager.go:414] "containerMap: removing container" podUID="947838c7-f014-4695-b9b8-04e78bfc5a21" containerName="revoke-keys"
Oct 30 21:58:42.966318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966280    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947838c7-f014-4695-b9b8-04e78bfc5a21" containerName="revoke-keys"
Oct 30 21:58:42.966318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966291    4410 cpu_manager.go:414] "containerMap: removing container" podUID="45940d30-4d56-4c45-90be-14c4b5006935" containerName="revoke-keys"
Oct 30 21:58:42.966318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966296    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45940d30-4d56-4c45-90be-14c4b5006935" containerName="revoke-keys"
Oct 30 21:58:42.966408 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966372    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="947838c7-f014-4695-b9b8-04e78bfc5a21" containerName="revoke-keys"
Oct 30 21:58:42.966408 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.966380    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="45940d30-4d56-4c45-90be-14c4b5006935" containerName="revoke-keys"
Oct 30 21:58:42.970019 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.970003    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:42.992803 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.992768    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmmm\" (UniqueName: \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:42.992909 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.992871    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:42.992957 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:42.992942    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.024868 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.024849    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:58:43.027036 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.027005    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"]
Oct 30 21:58:43.073951 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.073929    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:43.073951 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.073931    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:43.074083 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.073939    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:43.094088 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.094067    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmmm\" (UniqueName: \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.094180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.094118    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.094180 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.094165    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.094715 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.094692    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.096463 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.096437    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.150888 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.150861    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmmm\" (UniqueName: \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\") pod \"maas-api-revoke-sub-2dc19571-jzxnb\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.283699 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.283635    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:43.653712 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:43.653679    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"]
Oct 30 21:58:43.657118 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:43.657079    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd933ccfc_20f8_4805_af85_6edd50130923.slice/crio-2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c WatchSource:0}: Error finding container 2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c: Status 404 returned error can't find the container with id 2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c
Oct 30 21:58:44.277896 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:44.277815    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb" event={"ID":"d933ccfc-20f8-4805-af85-6edd50130923","Type":"ContainerStarted","Data":"f73a46e56a85a719402d5443813799a4df3909f86c0f8efdf93fda2cc3160219"}
Oct 30 21:58:44.277896 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:44.277879    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb" event={"ID":"d933ccfc-20f8-4805-af85-6edd50130923","Type":"ContainerStarted","Data":"2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c"}
Oct 30 21:58:44.390642 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:44.390591    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb" podStartSLOduration=2.390577571 podStartE2EDuration="2.390577571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:44.390443196 +0000 UTC m=+977.479180999" watchObservedRunningTime="2026-10-30 21:58:44.390577571 +0000 UTC m=+977.479315374"
Oct 30 21:58:45.287015 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:45.286977    4410 generic.go:356] "Generic (PLEG): container finished" podID="d933ccfc-20f8-4805-af85-6edd50130923" containerID="f73a46e56a85a719402d5443813799a4df3909f86c0f8efdf93fda2cc3160219" exitCode=0
Oct 30 21:58:45.287471 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:45.287029    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb" event={"ID":"d933ccfc-20f8-4805-af85-6edd50130923","Type":"ContainerDied","Data":"f73a46e56a85a719402d5443813799a4df3909f86c0f8efdf93fda2cc3160219"}
Oct 30 21:58:45.644475 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:45.644439    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:46.427361 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.427336    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:46.527462 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.527434    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\" (UniqueName: \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\") pod \"d933ccfc-20f8-4805-af85-6edd50130923\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") "
Oct 30 21:58:46.527570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.527467    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\") pod \"d933ccfc-20f8-4805-af85-6edd50130923\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") "
Oct 30 21:58:46.527570 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.527492    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\") pod \"d933ccfc-20f8-4805-af85-6edd50130923\" (UID: \"d933ccfc-20f8-4805-af85-6edd50130923\") "
Oct 30 21:58:46.527877 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.527848    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca" pod "d933ccfc-20f8-4805-af85-6edd50130923" (UID: "d933ccfc-20f8-4805-af85-6edd50130923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:46.528001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.527911    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:46.529588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.529566    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm" pod "d933ccfc-20f8-4805-af85-6edd50130923" (UID: "d933ccfc-20f8-4805-af85-6edd50130923"). InnerVolumeSpecName "kube-api-access-npmmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:46.529588 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.529576    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca" pod "d933ccfc-20f8-4805-af85-6edd50130923" (UID: "d933ccfc-20f8-4805-af85-6edd50130923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:46.628736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.628667    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmmm\" (UniqueName: \"kubernetes.io/projected/d933ccfc-20f8-4805-af85-6edd50130923-kube-api-access-npmmm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:46.628736 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:46.628694    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d933ccfc-20f8-4805-af85-6edd50130923-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.298389 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.298312    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"
Oct 30 21:58:47.298537 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.298310    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb" event={"ID":"d933ccfc-20f8-4805-af85-6edd50130923","Type":"ContainerDied","Data":"2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c"}
Oct 30 21:58:47.298537 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.298436    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f96c81ecda6b9b83e69bb9b14424c49219ab033061fccc9ad28ae14bafa629c"
Oct 30 21:58:47.463303 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.463273    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"]
Oct 30 21:58:47.463667 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.463624    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d933ccfc-20f8-4805-af85-6edd50130923" containerName="revoke-keys"
Oct 30 21:58:47.463667 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.463636    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d933ccfc-20f8-4805-af85-6edd50130923" containerName="revoke-keys"
Oct 30 21:58:47.463749 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.463739    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d933ccfc-20f8-4805-af85-6edd50130923" containerName="revoke-keys"
Oct 30 21:58:47.466801 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.466765    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.519124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.519095    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:47.519124 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.519098    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:47.519312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.519128    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:47.519312 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.519260    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:58:47.535450 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.535422    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.535599 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.535468    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxjl\" (UniqueName: \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.535599 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.535562    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.542959 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.542927    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"]
Oct 30 21:58:47.637121 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.637036    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxjl\" (UniqueName: \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.637121 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.637094    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.637339 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.637194    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.637760 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.637728    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.639581 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.639557    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.694754 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.694727    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxjl\" (UniqueName: \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\") pod \"maas-api-revoke-sub-163b6ea4-rnqv5\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.771000 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.770969    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"]
Oct 30 21:58:47.774422 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.774406    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.779819 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.779799    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:47.838353 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.838307    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"]
Oct 30 21:58:47.839927 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.838921    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8f5\" (UniqueName: \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.839927 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.839052    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.839927 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.839130    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.940222 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.940192    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.940420 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.940241    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8f5\" (UniqueName: \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.940420 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.940317    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.941051 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.941030    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.942573 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.942552    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:47.999176 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:47.999141    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8f5\" (UniqueName: \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\") pod \"maas-api-revoke-sub-9bc2636e-k7wfd\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:48.087980 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:48.087952    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:48.152412 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:48.152383    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"]
Oct 30 21:58:48.153419 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:48.153388    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b0ca99_e0ab_45ee_b94c_2dff66dd106e.slice/crio-96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2 WatchSource:0}: Error finding container 96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2: Status 404 returned error can't find the container with id 96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2
Oct 30 21:58:48.304828 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:48.304716    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5" event={"ID":"34b0ca99-e0ab-45ee-b94c-2dff66dd106e","Type":"ContainerStarted","Data":"96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2"}
Oct 30 21:58:48.467685 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:48.467662    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"]
Oct 30 21:58:48.469368 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:48.469343    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680c635c_4ce2_414b_a78e_d05903df01d7.slice/crio-69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1 WatchSource:0}: Error finding container 69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1: Status 404 returned error can't find the container with id 69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1
Oct 30 21:58:49.311100 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:49.311071    4410 generic.go:356] "Generic (PLEG): container finished" podID="680c635c-4ce2-414b-a78e-d05903df01d7" containerID="61ddac12c296cf274af6ae317ac072ca74c0ba982604631c680a07524512f73c" exitCode=0
Oct 30 21:58:49.311233 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:49.311154    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd" event={"ID":"680c635c-4ce2-414b-a78e-d05903df01d7","Type":"ContainerDied","Data":"61ddac12c296cf274af6ae317ac072ca74c0ba982604631c680a07524512f73c"}
Oct 30 21:58:49.311233 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:49.311213    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd" event={"ID":"680c635c-4ce2-414b-a78e-d05903df01d7","Type":"ContainerStarted","Data":"69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1"}
Oct 30 21:58:49.312630 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:49.312603    4410 generic.go:356] "Generic (PLEG): container finished" podID="34b0ca99-e0ab-45ee-b94c-2dff66dd106e" containerID="d9174e240318b5600ec8baf256fbec3e786da43fd2f4ed3fd2311d0aed1285bc" exitCode=0
Oct 30 21:58:49.312720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:49.312642    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5" event={"ID":"34b0ca99-e0ab-45ee-b94c-2dff66dd106e","Type":"ContainerDied","Data":"d9174e240318b5600ec8baf256fbec3e786da43fd2f4ed3fd2311d0aed1285bc"}
Oct 30 21:58:50.446660 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.446626    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:50.478309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.478284    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:50.521965 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:58:50.521931    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:58:50.564206 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.564175    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:50.571073 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571018    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\") pod \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") "
Oct 30 21:58:50.571073 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571055    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\") pod \"680c635c-4ce2-414b-a78e-d05903df01d7\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") "
Oct 30 21:58:50.571213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571111    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\" (UniqueName: \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\") pod \"680c635c-4ce2-414b-a78e-d05903df01d7\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") "
Oct 30 21:58:50.571213 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571192    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\") pod \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") "
Oct 30 21:58:50.571308 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571269    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\" (UniqueName: \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\") pod \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\" (UID: \"34b0ca99-e0ab-45ee-b94c-2dff66dd106e\") "
Oct 30 21:58:50.571366 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571305    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\") pod \"680c635c-4ce2-414b-a78e-d05903df01d7\" (UID: \"680c635c-4ce2-414b-a78e-d05903df01d7\") "
Oct 30 21:58:50.571761 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571703    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca" pod "34b0ca99-e0ab-45ee-b94c-2dff66dd106e" (UID: "34b0ca99-e0ab-45ee-b94c-2dff66dd106e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:50.571761 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.571753    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca" pod "680c635c-4ce2-414b-a78e-d05903df01d7" (UID: "680c635c-4ce2-414b-a78e-d05903df01d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:50.573147 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.573123    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca" pod "34b0ca99-e0ab-45ee-b94c-2dff66dd106e" (UID: "34b0ca99-e0ab-45ee-b94c-2dff66dd106e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:50.573336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.573316    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5" pod "680c635c-4ce2-414b-a78e-d05903df01d7" (UID: "680c635c-4ce2-414b-a78e-d05903df01d7"). InnerVolumeSpecName "kube-api-access-rj8f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:50.573466 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.573445    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca" pod "680c635c-4ce2-414b-a78e-d05903df01d7" (UID: "680c635c-4ce2-414b-a78e-d05903df01d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:50.573863 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.573843    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl" pod "34b0ca99-e0ab-45ee-b94c-2dff66dd106e" (UID: "34b0ca99-e0ab-45ee-b94c-2dff66dd106e"). InnerVolumeSpecName "kube-api-access-pfxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:50.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672503    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:50.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672540    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxjl\" (UniqueName: \"kubernetes.io/projected/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-kube-api-access-pfxjl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:50.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672555    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:50.672811 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672569    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b0ca99-e0ab-45ee-b94c-2dff66dd106e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:50.672811 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672583    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680c635c-4ce2-414b-a78e-d05903df01d7-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:50.672811 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:50.672596    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8f5\" (UniqueName: \"kubernetes.io/projected/680c635c-4ce2-414b-a78e-d05903df01d7-kube-api-access-rj8f5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:51.323925 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.323846    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"
Oct 30 21:58:51.323925 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.323861    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd" event={"ID":"680c635c-4ce2-414b-a78e-d05903df01d7","Type":"ContainerDied","Data":"69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1"}
Oct 30 21:58:51.323925 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.323912    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69399440b6061c2de51984e52628754f8c0d27141563c961def912f04e0156c1"
Oct 30 21:58:51.325178 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.325159    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"
Oct 30 21:58:51.325178 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.325164    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5" event={"ID":"34b0ca99-e0ab-45ee-b94c-2dff66dd106e","Type":"ContainerDied","Data":"96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2"}
Oct 30 21:58:51.325336 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:51.325198    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c9494fdd7ad635e2210eedc2657e8399a783c58feba983ca6dae55da613ab2"
Oct 30 21:58:53.228880 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.228849    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 21:58:53.229245 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229201    4410 cpu_manager.go:414] "containerMap: removing container" podUID="680c635c-4ce2-414b-a78e-d05903df01d7" containerName="revoke-keys"
Oct 30 21:58:53.229245 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229212    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680c635c-4ce2-414b-a78e-d05903df01d7" containerName="revoke-keys"
Oct 30 21:58:53.229245 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229242    4410 cpu_manager.go:414] "containerMap: removing container" podUID="34b0ca99-e0ab-45ee-b94c-2dff66dd106e" containerName="revoke-keys"
Oct 30 21:58:53.229346 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229247    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b0ca99-e0ab-45ee-b94c-2dff66dd106e" containerName="revoke-keys"
Oct 30 21:58:53.229346 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229306    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b0ca99-e0ab-45ee-b94c-2dff66dd106e" containerName="revoke-keys"
Oct 30 21:58:53.229346 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.229316    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="680c635c-4ce2-414b-a78e-d05903df01d7" containerName="revoke-keys"
Oct 30 21:58:53.232515 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.232498    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.285841 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.285816    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\""
Oct 30 21:58:53.297165 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297139    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297256 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297182    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297293 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297255    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297331 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297291    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297331 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297310    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzc4\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297391 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297332    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297430 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297398    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297464 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297442    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.297523 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.297502    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.337542 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.337519    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 21:58:53.398720 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.398692    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.398741    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399140    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzc4\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399207    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399241    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399297    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399344    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399505    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399573    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399875    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400143 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.399912    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.400727 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.400351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.406268 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.401144    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.406268 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.401414    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.406268 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.403310    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.407691 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.407662    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.456562 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.456528    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.456712 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.456693    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzc4\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.544753 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.544677    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:53.653152 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.653123    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"]
Oct 30 21:58:53.656836 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.656814    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.703067 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.703033    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5qg\" (UniqueName: \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.703218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.703091    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.703218 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.703116    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.710666 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.710537    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:53.710666 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.710626    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:53.710961 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.710826    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:53.711019 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.710980    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 21:58:53.713886 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.713861    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"]
Oct 30 21:58:53.804260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.804192    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.804260 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.804232    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.804436 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.804311    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5qg\" (UniqueName: \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.804939 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.804917    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.806521 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.806502    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.861716 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.861686    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5qg\" (UniqueName: \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\") pod \"maas-api-revoke-sub-be618162-l9f2v\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.971713 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.971678    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:53.977738 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.977719    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 21:58:53.982085 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:53.982059    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6925356d_9e1e_44ef_9635_d6c9ad0ed416.slice/crio-9fea44081d97b148036bdc0fff627be696fac521a8f73855ab97a2cbf218250e WatchSource:0}: Error finding container 9fea44081d97b148036bdc0fff627be696fac521a8f73855ab97a2cbf218250e: Status 404 returned error can't find the container with id 9fea44081d97b148036bdc0fff627be696fac521a8f73855ab97a2cbf218250e
Oct 30 21:58:53.984647 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.984611    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:53.984723 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.984698    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:53.984797 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:53.984753    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:54.343846 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:54.343806    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" event={"ID":"6925356d-9e1e-44ef-9635-d6c9ad0ed416","Type":"ContainerStarted","Data":"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"}
Oct 30 21:58:54.344238 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:54.343877    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" event={"ID":"6925356d-9e1e-44ef-9635-d6c9ad0ed416","Type":"ContainerStarted","Data":"9fea44081d97b148036bdc0fff627be696fac521a8f73855ab97a2cbf218250e"}
Oct 30 21:58:54.349390 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:54.349368    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"]
Oct 30 21:58:54.351303 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:58:54.351276    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa7cc05_0020_4d59_86ec_ae9e4590c01a.slice/crio-4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3 WatchSource:0}: Error finding container 4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3: Status 404 returned error can't find the container with id 4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3
Oct 30 21:58:54.468796 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:54.466058    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podStartSLOduration=1.466038895 podStartE2EDuration="1.466038895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:54.465297308 +0000 UTC m=+987.554035112" watchObservedRunningTime="2026-10-30 21:58:54.466038895 +0000 UTC m=+987.554776699"
Oct 30 21:58:54.552901 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:54.552865    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:58:55.374370 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.374335    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v" event={"ID":"afa7cc05-0020-4d59-86ec-ae9e4590c01a","Type":"ContainerStarted","Data":"ab06169b6dec4a20b374494feadfdabd95426671e1e859c97220750fe4c07cf9"}
Oct 30 21:58:55.374770 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.374418    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v" event={"ID":"afa7cc05-0020-4d59-86ec-ae9e4590c01a","Type":"ContainerStarted","Data":"4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3"}
Oct 30 21:58:55.396318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.396288    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:55.494921 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.494867    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v" podStartSLOduration=2.494850556 podStartE2EDuration="2.494850556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:55.494570336 +0000 UTC m=+988.583308140" watchObservedRunningTime="2026-10-30 21:58:55.494850556 +0000 UTC m=+988.583588353"
Oct 30 21:58:55.570745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.570709    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:58:55.570913 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:55.570812    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:58:56.380855 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:56.380823    4410 generic.go:356] "Generic (PLEG): container finished" podID="afa7cc05-0020-4d59-86ec-ae9e4590c01a" containerID="ab06169b6dec4a20b374494feadfdabd95426671e1e859c97220750fe4c07cf9" exitCode=0
Oct 30 21:58:56.381288 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:56.380916    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v" event={"ID":"afa7cc05-0020-4d59-86ec-ae9e4590c01a","Type":"ContainerDied","Data":"ab06169b6dec4a20b374494feadfdabd95426671e1e859c97220750fe4c07cf9"}
Oct 30 21:58:56.553214 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:56.553180    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:58:56.553455 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:56.553429    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:58:57.530038 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.530014    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:57.570237 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.570197    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:58:57.570377 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.570258    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:58:57.674002 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.673046    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\") pod \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") "
Oct 30 21:58:57.674002 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.673128    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\" (UniqueName: \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\") pod \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") "
Oct 30 21:58:57.674002 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.673207    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\") pod \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\" (UID: \"afa7cc05-0020-4d59-86ec-ae9e4590c01a\") "
Oct 30 21:58:57.675001 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.674707    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca" pod "afa7cc05-0020-4d59-86ec-ae9e4590c01a" (UID: "afa7cc05-0020-4d59-86ec-ae9e4590c01a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:57.677193 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.677164    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg" pod "afa7cc05-0020-4d59-86ec-ae9e4590c01a" (UID: "afa7cc05-0020-4d59-86ec-ae9e4590c01a"). InnerVolumeSpecName "kube-api-access-8f5qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:57.677800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.677279    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca" pod "afa7cc05-0020-4d59-86ec-ae9e4590c01a" (UID: "afa7cc05-0020-4d59-86ec-ae9e4590c01a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:57.774448 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.774416    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:57.774448 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.774444    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa7cc05-0020-4d59-86ec-ae9e4590c01a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:57.774448 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:57.774453    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5qg\" (UniqueName: \"kubernetes.io/projected/afa7cc05-0020-4d59-86ec-ae9e4590c01a-kube-api-access-8f5qg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:58.391744 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:58.391714    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"
Oct 30 21:58:58.391930 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:58.391715    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v" event={"ID":"afa7cc05-0020-4d59-86ec-ae9e4590c01a","Type":"ContainerDied","Data":"4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3"}
Oct 30 21:58:58.391930 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:58.391845    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4cde38ce3863221ef842dd8c6d017346f090c7f56a4d6ec8469d2c301ac8e3"
Oct 30 21:58:58.564799 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:58.561316    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:58:58.564799 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:58.561376    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:58:59.559800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:59.550955    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:58:59.559800 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:58:59.551020    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:59:00.172683 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:00.172643    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:00.613426 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:00.613344    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:59:00.613878 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:00.613847    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:59:00.614591 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:00.614566    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 21:59:02.168535 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:02.168494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:02.319202 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:02.319163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:02.482363 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:02.482273    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:05.246589 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:05.246543    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:06.577309 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:06.577269    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:10.086677 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:10.086644    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:13.931803 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:13.931744    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:14.138244 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:14.138211    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:14.933365 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:14.933325    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:18.911745 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:18.911707    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:19.740352 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:19.740315    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:20.522625 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:59:20.522589    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:59:24.502963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.502917    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 21:59:24.503338 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.503311    4410 cpu_manager.go:414] "containerMap: removing container" podUID="afa7cc05-0020-4d59-86ec-ae9e4590c01a" containerName="revoke-keys"
Oct 30 21:59:24.503338 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.503322    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa7cc05-0020-4d59-86ec-ae9e4590c01a" containerName="revoke-keys"
Oct 30 21:59:24.503414 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.503392    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa7cc05-0020-4d59-86ec-ae9e4590c01a" containerName="revoke-keys"
Oct 30 21:59:24.506580 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.506563    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.540768 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.540743    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.540910 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.540853    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv24w\" (UniqueName: \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.540910 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.540897    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.541029 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.541014    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.541071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.541048    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.541071 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.541065    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.564067 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.564038    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 21:59:24.609101 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.609068    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:24.610428 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.610400    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"e2e-conflict-a-9b1cb2d8-kserve-self-signed-certs\""
Oct 30 21:59:24.642378 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642351    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642493 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642384    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642493 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642423    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642493 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642481    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv24w\" (UniqueName: \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642641 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642525    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642641 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642615    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642847 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642825    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642960 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642871    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.642960 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.642946    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.644677 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.644655    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.644963 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.644944    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.699136 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.699102    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv24w\" (UniqueName: \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\") pod \"e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:24.767486 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.767398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:24.818091 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:24.818062    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:25.201762 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:25.201740    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 21:59:25.203686 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:59:25.203661    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf14351d_2e6a_472d_9f5a_3cac978ad7d0.slice/crio-7fb83c1ff5de6616c27538059fb0e706c1a8d33dd2661d45511b6f844c5e9474 WatchSource:0}: Error finding container 7fb83c1ff5de6616c27538059fb0e706c1a8d33dd2661d45511b6f844c5e9474: Status 404 returned error can't find the container with id 7fb83c1ff5de6616c27538059fb0e706c1a8d33dd2661d45511b6f844c5e9474
Oct 30 21:59:25.545457 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:25.545363    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" event={"ID":"df14351d-2e6a-472d-9f5a-3cac978ad7d0","Type":"ContainerStarted","Data":"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"}
Oct 30 21:59:25.545457 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:25.545416    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" event={"ID":"df14351d-2e6a-472d-9f5a-3cac978ad7d0","Type":"ContainerStarted","Data":"7fb83c1ff5de6616c27538059fb0e706c1a8d33dd2661d45511b6f844c5e9474"}
Oct 30 21:59:25.663438 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:25.663388    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" podStartSLOduration=1.6633755639999999 podStartE2EDuration="1.663375564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:25.661147138 +0000 UTC m=+1018.749884941" watchObservedRunningTime="2026-10-30 21:59:25.663375564 +0000 UTC m=+1018.752113364"
Oct 30 21:59:25.819302 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:25.819227    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:29.749926 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:29.749890    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.584539 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:34.584501    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.818957 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:34.818926    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:34.831533 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:34.831513    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:35.609398 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:35.609368    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 21:59:39.736568 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:39.736519    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:40.522843 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.522804    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 21:59:40.526284 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.526264    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603239 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603201    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603385 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603253    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603440 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603370    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603440 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603411    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603531 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603516    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603566 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603550    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zq85\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603683 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603668    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603752 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603717    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.603825 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.603791    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.631341 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.631315    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 21:59:40.704396 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704366    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704537 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704400    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zq85\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704585 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704633 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704597    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704633 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704617    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704731 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704661    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704731 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704707    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704887 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704754    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.704887 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.704799    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.705069 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.705050    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.705135 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.705088    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.705189 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.705148    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.705234 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.705189    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.706192 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.706165    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.706998 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.706975    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.707338 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.707319    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.760011 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.759982    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.760436 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.760414    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zq85\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\") pod \"maas-default-gateway-openshift-default-694db55c87-bm2hx\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:40.839076 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:40.839016    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:41.265223 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.265198    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 21:59:41.267027 ip-10-0-128-19 kubenswrapper[4410]: W1030 21:59:41.267000    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f313244_2f2a_40e1_8775_8146400ed562.slice/crio-1dfad838a334559175b2dbf3f0faf51a97b08f909231c26a51589023566bc8ed WatchSource:0}: Error finding container 1dfad838a334559175b2dbf3f0faf51a97b08f909231c26a51589023566bc8ed: Status 404 returned error can't find the container with id 1dfad838a334559175b2dbf3f0faf51a97b08f909231c26a51589023566bc8ed
Oct 30 21:59:41.269255 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.269227    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:41.269318 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.269291    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:41.269363 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.269318    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:41.636401 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.636312    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" event={"ID":"8f313244-2f2a-40e1-8775-8146400ed562","Type":"ContainerStarted","Data":"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"}
Oct 30 21:59:41.636401 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.636371    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" event={"ID":"8f313244-2f2a-40e1-8775-8146400ed562","Type":"ContainerStarted","Data":"1dfad838a334559175b2dbf3f0faf51a97b08f909231c26a51589023566bc8ed"}
Oct 30 21:59:41.753343 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.753271    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podStartSLOduration=1.753254399 podStartE2EDuration="1.753254399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:41.751942982 +0000 UTC m=+1034.840680787" watchObservedRunningTime="2026-10-30 21:59:41.753254399 +0000 UTC m=+1034.841992205"
Oct 30 21:59:41.839440 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:41.839405    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:42.841765 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:42.841731    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:42.842166 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:42.841823    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:43.839531 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:43.839499    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:43.839715 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:43.839549    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:44.785106 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:44.785068    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:44.840129 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:44.840096    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:44.840356 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:44.840157    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:45.840135 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:45.840097    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:45.840501 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:45.840165    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:46.839551 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:46.839514    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:46.839718 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:46.839595    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:47.839934 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:47.839896    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:47.840321 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:47.839974    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:48.839616 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:48.839581    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:59:48.839807 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:48.839637    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:59:49.730403 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:49.730367    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:49.843460 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:49.843432    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:49.843713 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:49.843685    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:49.844441 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:49.844425    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 21:59:50.521636 ip-10-0-128-19 kubenswrapper[4410]: E1030 21:59:50.521603    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 21:59:54.457049 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:54.457013    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:59.241467 ip-10-0-128-19 kubenswrapper[4410]: I1030 21:59:59.241420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:00.169528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.169496    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:00.173100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.173078    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:00.173242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.173210    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.176379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.176359    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.179934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.179913    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:00.183140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.183124    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.215127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215105    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.215245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7qj\" (UniqueName: \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.215245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215223    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5x62\" (UniqueName: \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.215335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215254    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhf2s\" (UniqueName: \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.215335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215286    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.215423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.215404    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.221862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.221834    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:00.226986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.226968    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.227094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.227029    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.227094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.227066    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:00:00.227094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.227068    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.268902 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.268878    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:00.277229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.277201    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:00.280546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.280521    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:00.316489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316458    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.316725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316698    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7qj\" (UniqueName: \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.316879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316755    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5x62\" (UniqueName: \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.316879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316800    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhf2s\" (UniqueName: \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.316879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316829    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8zp\" (UniqueName: \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.316879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316877    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.317140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.316957    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.317140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.317093    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.318125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.318096    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.318293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.318243    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.318863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.318840    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.322085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.322053    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:00.374463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.374437    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7qj\" (UniqueName: \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.374642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.374620    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5x62\" (UniqueName: \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\") pod \"maas-api-key-cleanup-29889960-jthnp\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.374707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.374662    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhf2s\" (UniqueName: \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.402853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.402826    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:00.417945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.417918    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.418048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.417963    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8zp\" (UniqueName: \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.418517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.418499    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.475701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.475621    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8zp\" (UniqueName: \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.487436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.487407    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:00.494740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.494721    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:00.502807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.502758    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:00.538367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.538336    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:00.880997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.880973    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:00.882649 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:00:00.882613    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63192f50_4032_4e27_9153_7714bbf330dd.slice/crio-f9633032f01fc2ef9df113d24d2401976aced8f312c5095b6ce464e004cc261b WatchSource:0}: Error finding container f9633032f01fc2ef9df113d24d2401976aced8f312c5095b6ce464e004cc261b: Status 404 returned error can't find the container with id f9633032f01fc2ef9df113d24d2401976aced8f312c5095b6ce464e004cc261b
Oct 30 22:00:00.926927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.926904    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:00.931374 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:00:00.931347    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f47983_12ba_4ff8_97c0_162443bae1a6.slice/crio-23b659faad01cee394f25340b0bb82d7d0370e649e9443430be9d81fda4bfaa8 WatchSource:0}: Error finding container 23b659faad01cee394f25340b0bb82d7d0370e649e9443430be9d81fda4bfaa8: Status 404 returned error can't find the container with id 23b659faad01cee394f25340b0bb82d7d0370e649e9443430be9d81fda4bfaa8
Oct 30 22:00:00.967924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.967899    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:00.968931 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:00:00.968906    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249fd36d_6780_4253_9cd9_298061a1b215.slice/crio-3f1d72dc239ac28b1550c031a9348cf862904483161a9ed9d9ef9b5a2d1eb4d7 WatchSource:0}: Error finding container 3f1d72dc239ac28b1550c031a9348cf862904483161a9ed9d9ef9b5a2d1eb4d7: Status 404 returned error can't find the container with id 3f1d72dc239ac28b1550c031a9348cf862904483161a9ed9d9ef9b5a2d1eb4d7
Oct 30 22:00:00.979064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:00.979044    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:00.981386 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:00:00.981364    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efe4dbe_7cea_4edd_8d14_264d2b88ea9b.slice/crio-dfa74f78b35ea6d1af18fb6f77c51b3dcc4a94f83ff28e2660b6909661f094e6 WatchSource:0}: Error finding container dfa74f78b35ea6d1af18fb6f77c51b3dcc4a94f83ff28e2660b6909661f094e6: Status 404 returned error can't find the container with id dfa74f78b35ea6d1af18fb6f77c51b3dcc4a94f83ff28e2660b6909661f094e6
Oct 30 22:00:01.766112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.766079    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerStarted","Data":"55bfb8fc2461d2f406a6cb3ef97d211e3a5d79278f72840f746453a3dc1812f5"}
Oct 30 22:00:01.766490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.766145    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerStarted","Data":"23b659faad01cee394f25340b0bb82d7d0370e649e9443430be9d81fda4bfaa8"}
Oct 30 22:00:01.767860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.767833    4410 generic.go:356] "Generic (PLEG): container finished" podID="63192f50-4032-4e27-9153-7714bbf330dd" containerID="6f15ca492ff26cf220c682d047d7738b19fa3b1b84785d275fa0c388412297cc" exitCode=60
Oct 30 22:00:01.767982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.767901    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" event={"ID":"63192f50-4032-4e27-9153-7714bbf330dd","Type":"ContainerDied","Data":"6f15ca492ff26cf220c682d047d7738b19fa3b1b84785d275fa0c388412297cc"}
Oct 30 22:00:01.767982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.767954    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" event={"ID":"63192f50-4032-4e27-9153-7714bbf330dd","Type":"ContainerStarted","Data":"f9633032f01fc2ef9df113d24d2401976aced8f312c5095b6ce464e004cc261b"}
Oct 30 22:00:01.768238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.768220    4410 scope.go:122] "RemoveContainer" containerID="6f15ca492ff26cf220c682d047d7738b19fa3b1b84785d275fa0c388412297cc"
Oct 30 22:00:01.770377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.770340    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerStarted","Data":"b8a46a6f78ff22bd50523559f06462ba0ec1809f34cfea75873f02fa2650e22e"}
Oct 30 22:00:01.770496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.770391    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerStarted","Data":"3f1d72dc239ac28b1550c031a9348cf862904483161a9ed9d9ef9b5a2d1eb4d7"}
Oct 30 22:00:01.772492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.772466    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerStarted","Data":"928cbe6c945c14bfe3b5552f7026ba4dd084cfabfba8afb3aacff7a0af833c85"}
Oct 30 22:00:01.772595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.772513    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerStarted","Data":"dfa74f78b35ea6d1af18fb6f77c51b3dcc4a94f83ff28e2660b6909661f094e6"}
Oct 30 22:00:01.881762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.881711    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" podStartSLOduration=1.8816867849999999 podStartE2EDuration="1.881686785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.878299711 +0000 UTC m=+1054.967037513" watchObservedRunningTime="2026-10-30 22:00:01.881686785 +0000 UTC m=+1054.970424590"
Oct 30 22:00:01.967376 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:01.967343    4410 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63192f50_4032_4e27_9153_7714bbf330dd.slice/crio-96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:01.992682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:01.992632    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" podStartSLOduration=1.992618712 podStartE2EDuration="1.992618712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.990635533 +0000 UTC m=+1055.079373339" watchObservedRunningTime="2026-10-30 22:00:01.992618712 +0000 UTC m=+1055.081356515"
Oct 30 22:00:02.217954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.217899    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" podStartSLOduration=2.217884415 podStartE2EDuration="2.217884415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.215480257 +0000 UTC m=+1055.304218072" watchObservedRunningTime="2026-10-30 22:00:02.217884415 +0000 UTC m=+1055.306622256"
Oct 30 22:00:02.778896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.778861    4410 generic.go:356] "Generic (PLEG): container finished" podID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerID="55bfb8fc2461d2f406a6cb3ef97d211e3a5d79278f72840f746453a3dc1812f5" exitCode=60
Oct 30 22:00:02.779277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.778951    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerDied","Data":"55bfb8fc2461d2f406a6cb3ef97d211e3a5d79278f72840f746453a3dc1812f5"}
Oct 30 22:00:02.779277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.779121    4410 scope.go:122] "RemoveContainer" containerID="55bfb8fc2461d2f406a6cb3ef97d211e3a5d79278f72840f746453a3dc1812f5"
Oct 30 22:00:02.780908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.780889    4410 generic.go:356] "Generic (PLEG): container finished" podID="63192f50-4032-4e27-9153-7714bbf330dd" containerID="96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665" exitCode=60
Oct 30 22:00:02.780991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.780956    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" event={"ID":"63192f50-4032-4e27-9153-7714bbf330dd","Type":"ContainerDied","Data":"96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665"}
Oct 30 22:00:02.781044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.781005    4410 scope.go:122] "RemoveContainer" containerID="6f15ca492ff26cf220c682d047d7738b19fa3b1b84785d275fa0c388412297cc"
Oct 30 22:00:02.781191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.781174    4410 scope.go:122] "RemoveContainer" containerID="96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665"
Oct 30 22:00:02.781964 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:02.781804    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m_odh-ai-gateway-infra(63192f50-4032-4e27-9153-7714bbf330dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" podUID="63192f50-4032-4e27-9153-7714bbf330dd"
Oct 30 22:00:02.782515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.782491    4410 generic.go:356] "Generic (PLEG): container finished" podID="249fd36d-6780-4253-9cd9-298061a1b215" containerID="b8a46a6f78ff22bd50523559f06462ba0ec1809f34cfea75873f02fa2650e22e" exitCode=60
Oct 30 22:00:02.782615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.782524    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerDied","Data":"b8a46a6f78ff22bd50523559f06462ba0ec1809f34cfea75873f02fa2650e22e"}
Oct 30 22:00:02.782746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.782731    4410 scope.go:122] "RemoveContainer" containerID="b8a46a6f78ff22bd50523559f06462ba0ec1809f34cfea75873f02fa2650e22e"
Oct 30 22:00:02.784187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.784168    4410 generic.go:356] "Generic (PLEG): container finished" podID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerID="928cbe6c945c14bfe3b5552f7026ba4dd084cfabfba8afb3aacff7a0af833c85" exitCode=60
Oct 30 22:00:02.784276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.784222    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerDied","Data":"928cbe6c945c14bfe3b5552f7026ba4dd084cfabfba8afb3aacff7a0af833c85"}
Oct 30 22:00:02.784396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:02.784384    4410 scope.go:122] "RemoveContainer" containerID="928cbe6c945c14bfe3b5552f7026ba4dd084cfabfba8afb3aacff7a0af833c85"
Oct 30 22:00:03.791815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.791783    4410 generic.go:356] "Generic (PLEG): container finished" podID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerID="30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554" exitCode=60
Oct 30 22:00:03.792204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.791868    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerDied","Data":"30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554"}
Oct 30 22:00:03.792204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.791925    4410 scope.go:122] "RemoveContainer" containerID="928cbe6c945c14bfe3b5552f7026ba4dd084cfabfba8afb3aacff7a0af833c85"
Oct 30 22:00:03.792204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.792141    4410 scope.go:122] "RemoveContainer" containerID="30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554"
Oct 30 22:00:03.792454 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:03.792429    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc_odh-ai-gateway-infra(2efe4dbe-7cea-4edd-8d14-264d2b88ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b"
Oct 30 22:00:03.793867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.793846    4410 generic.go:356] "Generic (PLEG): container finished" podID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerID="bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d" exitCode=60
Oct 30 22:00:03.793954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.793927    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerDied","Data":"bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d"}
Oct 30 22:00:03.794181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.794165    4410 scope.go:122] "RemoveContainer" containerID="bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d"
Oct 30 22:00:03.794391 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:03.794368    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-jthnp_odh-ai-gateway-infra(82f47983-12ba-4ff8-97c0-162443bae1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6"
Oct 30 22:00:03.795651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.795635    4410 scope.go:122] "RemoveContainer" containerID="96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665"
Oct 30 22:00:03.795907 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:03.795842    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m_odh-ai-gateway-infra(63192f50-4032-4e27-9153-7714bbf330dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" podUID="63192f50-4032-4e27-9153-7714bbf330dd"
Oct 30 22:00:03.797356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.797336    4410 generic.go:356] "Generic (PLEG): container finished" podID="249fd36d-6780-4253-9cd9-298061a1b215" containerID="8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669" exitCode=60
Oct 30 22:00:03.797434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.797375    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerDied","Data":"8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669"}
Oct 30 22:00:03.797587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.797573    4410 scope.go:122] "RemoveContainer" containerID="8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669"
Oct 30 22:00:03.797792 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:03.797755    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6_odh-ai-gateway-infra(249fd36d-6780-4253-9cd9-298061a1b215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" podUID="249fd36d-6780-4253-9cd9-298061a1b215"
Oct 30 22:00:03.817327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.817310    4410 scope.go:122] "RemoveContainer" containerID="55bfb8fc2461d2f406a6cb3ef97d211e3a5d79278f72840f746453a3dc1812f5"
Oct 30 22:00:03.828789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:03.828754    4410 scope.go:122] "RemoveContainer" containerID="b8a46a6f78ff22bd50523559f06462ba0ec1809f34cfea75873f02fa2650e22e"
Oct 30 22:00:04.151471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:04.151437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:04.804116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:04.804085    4410 scope.go:122] "RemoveContainer" containerID="8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669"
Oct 30 22:00:04.804537 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:04.804366    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6_odh-ai-gateway-infra(249fd36d-6780-4253-9cd9-298061a1b215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" podUID="249fd36d-6780-4253-9cd9-298061a1b215"
Oct 30 22:00:04.805819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:04.805769    4410 scope.go:122] "RemoveContainer" containerID="30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554"
Oct 30 22:00:04.806046 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:04.806018    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc_odh-ai-gateway-infra(2efe4dbe-7cea-4edd-8d14-264d2b88ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b"
Oct 30 22:00:04.807265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:04.807239    4410 scope.go:122] "RemoveContainer" containerID="bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d"
Oct 30 22:00:04.807452 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:04.807432    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-jthnp_odh-ai-gateway-infra(82f47983-12ba-4ff8-97c0-162443bae1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6"
Oct 30 22:00:05.619960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.619931    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:00:05.624743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.624728    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676564    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676594    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676631    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676742    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676836    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676862    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.676940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676882    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.677002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676987    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b8717e-openshift-default-dockercfg-pqzrg\""
Oct 30 22:00:05.677040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.676997    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gv42\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.677084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.677045    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.729055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.729032    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:00:05.777884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.777858    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.777999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.777908    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.777999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.777945    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.777999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.777977    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.777999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.777997    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778017    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778073    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gv42\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778142    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778169    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778297    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778546    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778616    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778616    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.778910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.778889    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.780187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.780164    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.780419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.780403    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.836479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.836448    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.836828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.836665    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gv42\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:05.936796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:05.936736    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:06.362591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.362569    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:00:06.364263 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:00:06.364237    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7e3514_caca_4abb_8a59_8b1aed8cc9c2.slice/crio-ae0bcc130123846856927922831ef79d6e15b5607d966cec3d2d38feebd1fd1a WatchSource:0}: Error finding container ae0bcc130123846856927922831ef79d6e15b5607d966cec3d2d38feebd1fd1a: Status 404 returned error can't find the container with id ae0bcc130123846856927922831ef79d6e15b5607d966cec3d2d38feebd1fd1a
Oct 30 22:00:06.366617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.366585    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:06.366701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.366647    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:06.366701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.366675    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:06.818387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.818352    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" event={"ID":"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2","Type":"ContainerStarted","Data":"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"}
Oct 30 22:00:06.818541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.818408    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" event={"ID":"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2","Type":"ContainerStarted","Data":"ae0bcc130123846856927922831ef79d6e15b5607d966cec3d2d38feebd1fd1a"}
Oct 30 22:00:06.937870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.937820    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:06.994819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:06.994740    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podStartSLOduration=1.994726185 podStartE2EDuration="1.994726185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.992104542 +0000 UTC m=+1060.080842345" watchObservedRunningTime="2026-10-30 22:00:06.994726185 +0000 UTC m=+1060.083463989"
Oct 30 22:00:07.938597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:07.938567    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:07.939006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:07.938615    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:08.676059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:08.676022    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:08.938161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:08.938075    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:08.938161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:08.938135    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:09.082381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:09.082350    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:09.937338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:09.937302    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:09.937489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:09.937359    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:10.937627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:10.937592    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:10.938027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:10.937661    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:11.937485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:11.937449    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:11.937656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:11.937501    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:12.937343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:12.937310    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:00:12.937536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:12.937366    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 22:00:13.941621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:13.941590    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:13.941995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:13.941972    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:13.942542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:13.942528    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:00:13.947140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:13.947113    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:14.361267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:14.361185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:14.514894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:14.514849    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:15.012716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.012683    4410 scope.go:122] "RemoveContainer" containerID="30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554"
Oct 30 22:00:15.013212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.012909    4410 scope.go:122] "RemoveContainer" containerID="96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665"
Oct 30 22:00:15.867912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.867882    4410 generic.go:356] "Generic (PLEG): container finished" podID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerID="017720e4fbf01912a606adb89c0cd26a5ed70d431224e695f9607b42fac969f5" exitCode=60
Oct 30 22:00:15.868053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.867963    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerDied","Data":"017720e4fbf01912a606adb89c0cd26a5ed70d431224e695f9607b42fac969f5"}
Oct 30 22:00:15.868112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.868050    4410 scope.go:122] "RemoveContainer" containerID="30b893fd954f4d6fdb03c8dda0763c2819d2402c44f9e119d43430d1bc7e8554"
Oct 30 22:00:15.868425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.868404    4410 scope.go:122] "RemoveContainer" containerID="017720e4fbf01912a606adb89c0cd26a5ed70d431224e695f9607b42fac969f5"
Oct 30 22:00:15.868741 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:15.868714    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc_odh-ai-gateway-infra(2efe4dbe-7cea-4edd-8d14-264d2b88ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b"
Oct 30 22:00:15.870634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.870616    4410 generic.go:356] "Generic (PLEG): container finished" podID="63192f50-4032-4e27-9153-7714bbf330dd" containerID="bf50b4133ff1a49a63c6bb8486387f58872bb3c74c11be627bedee6f40cd98dc" exitCode=60
Oct 30 22:00:15.870724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.870689    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" event={"ID":"63192f50-4032-4e27-9153-7714bbf330dd","Type":"ContainerDied","Data":"bf50b4133ff1a49a63c6bb8486387f58872bb3c74c11be627bedee6f40cd98dc"}
Oct 30 22:00:15.871210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.871193    4410 scope.go:122] "RemoveContainer" containerID="bf50b4133ff1a49a63c6bb8486387f58872bb3c74c11be627bedee6f40cd98dc"
Oct 30 22:00:15.871391 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:15.871372    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m_odh-ai-gateway-infra(63192f50-4032-4e27-9153-7714bbf330dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" podUID="63192f50-4032-4e27-9153-7714bbf330dd"
Oct 30 22:00:15.941542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:15.941518    4410 scope.go:122] "RemoveContainer" containerID="96367bc0d61a5906c2b7bb542a100aaf4f07c5e46ab252d2336527402acc6665"
Oct 30 22:00:16.012242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:16.012208    4410 scope.go:122] "RemoveContainer" containerID="bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d"
Oct 30 22:00:16.012404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:16.012360    4410 scope.go:122] "RemoveContainer" containerID="8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669"
Oct 30 22:00:16.879327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:16.879298    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerStarted","Data":"f339e6bac84d3d75df4629d32afbf58f4f12edd0b7496e2ee122cf814e9ab935"}
Oct 30 22:00:16.882580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:16.882558    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerStarted","Data":"682071c8fbd4d9863fbe873858720d935fd799e10d3b63b3b5e8a846433e291d"}
Oct 30 22:00:16.994146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:16.994112    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:17.106812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.106715    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:17.180992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.180967    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:17.299368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.299344    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:17.303595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.303569    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\") pod \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") "
Oct 30 22:00:17.303708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.303684    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\" (UniqueName: \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\") pod \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\" (UID: \"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b\") "
Oct 30 22:00:17.303915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.303895    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca" pod "2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" (UID: "2efe4dbe-7cea-4edd-8d14-264d2b88ea9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:17.304045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.304033    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.305892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.305866    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s" pod "2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" (UID: "2efe4dbe-7cea-4edd-8d14-264d2b88ea9b"). InnerVolumeSpecName "kube-api-access-mhf2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:17.404464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.404434    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\") pod \"63192f50-4032-4e27-9153-7714bbf330dd\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") "
Oct 30 22:00:17.404647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.404586    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\" (UniqueName: \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\") pod \"63192f50-4032-4e27-9153-7714bbf330dd\" (UID: \"63192f50-4032-4e27-9153-7714bbf330dd\") "
Oct 30 22:00:17.404893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.404871    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca" pod "63192f50-4032-4e27-9153-7714bbf330dd" (UID: "63192f50-4032-4e27-9153-7714bbf330dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:17.405027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.405009    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhf2s\" (UniqueName: \"kubernetes.io/projected/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b-kube-api-access-mhf2s\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.405027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.405027    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63192f50-4032-4e27-9153-7714bbf330dd-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.406682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.406658    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj" pod "63192f50-4032-4e27-9153-7714bbf330dd" (UID: "63192f50-4032-4e27-9153-7714bbf330dd"). InnerVolumeSpecName "kube-api-access-tf7qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:17.506186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.506155    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7qj\" (UniqueName: \"kubernetes.io/projected/63192f50-4032-4e27-9153-7714bbf330dd-kube-api-access-tf7qj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.898014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.897985    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"
Oct 30 22:00:17.898494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.897989    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m" event={"ID":"63192f50-4032-4e27-9153-7714bbf330dd","Type":"ContainerDied","Data":"f9633032f01fc2ef9df113d24d2401976aced8f312c5095b6ce464e004cc261b"}
Oct 30 22:00:17.898494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.898171    4410 scope.go:122] "RemoveContainer" containerID="bf50b4133ff1a49a63c6bb8486387f58872bb3c74c11be627bedee6f40cd98dc"
Oct 30 22:00:17.900009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.899987    4410 generic.go:356] "Generic (PLEG): container finished" podID="249fd36d-6780-4253-9cd9-298061a1b215" containerID="682071c8fbd4d9863fbe873858720d935fd799e10d3b63b3b5e8a846433e291d" exitCode=60
Oct 30 22:00:17.900110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.900063    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerDied","Data":"682071c8fbd4d9863fbe873858720d935fd799e10d3b63b3b5e8a846433e291d"}
Oct 30 22:00:17.900400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.900382    4410 scope.go:122] "RemoveContainer" containerID="682071c8fbd4d9863fbe873858720d935fd799e10d3b63b3b5e8a846433e291d"
Oct 30 22:00:17.900658 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:17.900637    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6_odh-ai-gateway-infra(249fd36d-6780-4253-9cd9-298061a1b215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" podUID="249fd36d-6780-4253-9cd9-298061a1b215"
Oct 30 22:00:17.901649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.901629    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"
Oct 30 22:00:17.901746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.901653    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc" event={"ID":"2efe4dbe-7cea-4edd-8d14-264d2b88ea9b","Type":"ContainerDied","Data":"dfa74f78b35ea6d1af18fb6f77c51b3dcc4a94f83ff28e2660b6909661f094e6"}
Oct 30 22:00:17.903591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.903550    4410 generic.go:356] "Generic (PLEG): container finished" podID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerID="f339e6bac84d3d75df4629d32afbf58f4f12edd0b7496e2ee122cf814e9ab935" exitCode=60
Oct 30 22:00:17.903591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.903581    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerDied","Data":"f339e6bac84d3d75df4629d32afbf58f4f12edd0b7496e2ee122cf814e9ab935"}
Oct 30 22:00:17.903907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.903892    4410 scope.go:122] "RemoveContainer" containerID="f339e6bac84d3d75df4629d32afbf58f4f12edd0b7496e2ee122cf814e9ab935"
Oct 30 22:00:17.904099 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:17.904080    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-jthnp_odh-ai-gateway-infra(82f47983-12ba-4ff8-97c0-162443bae1a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6"
Oct 30 22:00:17.910017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.909994    4410 scope.go:122] "RemoveContainer" containerID="8d293eb7bd71ff922219aa3aef556182e19af9947113c5a5d9dc2b42ac20e669"
Oct 30 22:00:17.921684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.921667    4410 scope.go:122] "RemoveContainer" containerID="017720e4fbf01912a606adb89c0cd26a5ed70d431224e695f9607b42fac969f5"
Oct 30 22:00:17.929333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:17.929314    4410 scope.go:122] "RemoveContainer" containerID="bc5365cc3390710f3ac286659b85116ea28deae05f769ed3c6c97abdae37773d"
Oct 30 22:00:18.006874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.006844    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:18.182080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.182047    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:18.207963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.207937    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889960-qfj9m"]
Oct 30 22:00:18.249200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.249173    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:18.435263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.435184    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:18.439569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.439543    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889960-879lc"]
Oct 30 22:00:18.770634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:18.770562    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.023641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.023568    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" path="/var/lib/kubelet/pods/2efe4dbe-7cea-4edd-8d14-264d2b88ea9b/volumes"
Oct 30 22:00:19.024193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.024170    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63192f50-4032-4e27-9153-7714bbf330dd" path="/var/lib/kubelet/pods/63192f50-4032-4e27-9153-7714bbf330dd/volumes"
Oct 30 22:00:19.076586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.076556    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:19.104230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.104208    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:19.222407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222375    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\") pod \"249fd36d-6780-4253-9cd9-298061a1b215\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") "
Oct 30 22:00:19.222407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222412    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\") pod \"82f47983-12ba-4ff8-97c0-162443bae1a6\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") "
Oct 30 22:00:19.222631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222453    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\" (UniqueName: \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\") pod \"82f47983-12ba-4ff8-97c0-162443bae1a6\" (UID: \"82f47983-12ba-4ff8-97c0-162443bae1a6\") "
Oct 30 22:00:19.222631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222508    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\" (UniqueName: \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\") pod \"249fd36d-6780-4253-9cd9-298061a1b215\" (UID: \"249fd36d-6780-4253-9cd9-298061a1b215\") "
Oct 30 22:00:19.222865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222836    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca" pod "249fd36d-6780-4253-9cd9-298061a1b215" (UID: "249fd36d-6780-4253-9cd9-298061a1b215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.222865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222848    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca" pod "82f47983-12ba-4ff8-97c0-162443bae1a6" (UID: "82f47983-12ba-4ff8-97c0-162443bae1a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.223001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.222983    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249fd36d-6780-4253-9cd9-298061a1b215-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.223059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.223006    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f47983-12ba-4ff8-97c0-162443bae1a6-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.224678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.224649    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62" pod "82f47983-12ba-4ff8-97c0-162443bae1a6" (UID: "82f47983-12ba-4ff8-97c0-162443bae1a6"). InnerVolumeSpecName "kube-api-access-l5x62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.224678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.224670    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp" pod "249fd36d-6780-4253-9cd9-298061a1b215" (UID: "249fd36d-6780-4253-9cd9-298061a1b215"). InnerVolumeSpecName "kube-api-access-fs8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.257486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.257453    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.323622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.323553    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5x62\" (UniqueName: \"kubernetes.io/projected/82f47983-12ba-4ff8-97c0-162443bae1a6-kube-api-access-l5x62\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.323622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.323581    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs8zp\" (UniqueName: \"kubernetes.io/projected/249fd36d-6780-4253-9cd9-298061a1b215-kube-api-access-fs8zp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.924312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.924283    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"
Oct 30 22:00:19.924511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.924308    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp" event={"ID":"82f47983-12ba-4ff8-97c0-162443bae1a6","Type":"ContainerDied","Data":"23b659faad01cee394f25340b0bb82d7d0370e649e9443430be9d81fda4bfaa8"}
Oct 30 22:00:19.924511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.924424    4410 scope.go:122] "RemoveContainer" containerID="f339e6bac84d3d75df4629d32afbf58f4f12edd0b7496e2ee122cf814e9ab935"
Oct 30 22:00:19.925842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.925821    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6" event={"ID":"249fd36d-6780-4253-9cd9-298061a1b215","Type":"ContainerDied","Data":"3f1d72dc239ac28b1550c031a9348cf862904483161a9ed9d9ef9b5a2d1eb4d7"}
Oct 30 22:00:19.925842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.925835    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"
Oct 30 22:00:19.935155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:19.935135    4410 scope.go:122] "RemoveContainer" containerID="682071c8fbd4d9863fbe873858720d935fd799e10d3b63b3b5e8a846433e291d"
Oct 30 22:00:20.086883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:20.086857    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:20.090517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:20.090495    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889960-r59t6"]
Oct 30 22:00:20.244830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:20.244735    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:20.248679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:20.248650    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-jthnp"]
Oct 30 22:00:20.522371 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:20.522280    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:00:21.018692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:21.018661    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249fd36d-6780-4253-9cd9-298061a1b215" path="/var/lib/kubelet/pods/249fd36d-6780-4253-9cd9-298061a1b215/volumes"
Oct 30 22:00:21.019113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:21.019097    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" path="/var/lib/kubelet/pods/82f47983-12ba-4ff8-97c0-162443bae1a6/volumes"
Oct 30 22:00:23.676243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:23.675811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:27.133365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:27.133332    4410 scope.go:122] "RemoveContainer" containerID="e818788e39ea625fcec9d69dbb58be8c81fdf72d3a0b695618338c6048d5ef5d"
Oct 30 22:00:27.142341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:27.142319    4410 scope.go:122] "RemoveContainer" containerID="bed5ef76ac1b65367aea87421b7263da45ada7f574c823aa2c1f92153b90414b"
Oct 30 22:00:28.546719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:28.546679    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:32.274678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:32.274636    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:33.339580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:33.339539    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:38.150513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:38.150471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:42.149022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:42.148986    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:43.137219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:43.137189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:47.990860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:47.990820    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:50.523051 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:00:50.523017    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:00:52.878944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:52.878901    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:57.665282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:00:57.665250    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:02.476372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:02.476334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.305384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:07.305346    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.005364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005282    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:01:12.005907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005887    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005909    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005926    4410 cpu_manager.go:414] "containerMap: removing container" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005933    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005951    4410 cpu_manager.go:414] "containerMap: removing container" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005958    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005972    4410 cpu_manager.go:414] "containerMap: removing container" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005979    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.005991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005990    4410 cpu_manager.go:414] "containerMap: removing container" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.005998    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006017    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006025    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006051    4410 cpu_manager.go:414] "containerMap: removing container" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006058    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006069    4410 cpu_manager.go:414] "containerMap: removing container" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006076    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006094    4410 cpu_manager.go:414] "containerMap: removing container" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006101    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006206    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006219    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006228    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006240    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006249    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006259    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006268    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006280    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006288    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.006382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.006298    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:12.009855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.009834    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048608    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnkb\" (UniqueName: \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048651    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048758    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048902 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048854    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048937    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.048989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.048969    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.069787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.069754    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"distinct-w4-d2a3e0-kserve-self-signed-certs\""
Oct 30 22:01:12.121643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.121617    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:01:12.135703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.135676    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.151560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151531    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnkb\" (UniqueName: \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.151726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151577    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.151726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151629    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.151726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151666    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.151931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151740    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.151987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.151934    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.152388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.152360    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.153802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.152395    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.153802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.152654    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.155468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.155442    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.156412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.156387    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.208676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.208644    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnkb\" (UniqueName: \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\") pod \"distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.322189 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.322102    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:12.461452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.461405    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.711474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:12.711438    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:01:12.714205 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:01:12.714178    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e469d15_9f4a_4ac3_a422_b8509c7971f8.slice/crio-00150b309f267aa6eb7a6927a72db4b97217db766d966bd7e073878ffc74eb8b WatchSource:0}: Error finding container 00150b309f267aa6eb7a6927a72db4b97217db766d966bd7e073878ffc74eb8b: Status 404 returned error can't find the container with id 00150b309f267aa6eb7a6927a72db4b97217db766d966bd7e073878ffc74eb8b
Oct 30 22:01:13.227341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:13.227304    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" event={"ID":"2e469d15-9f4a-4ac3-a422-b8509c7971f8","Type":"ContainerStarted","Data":"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"}
Oct 30 22:01:13.227705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:13.227349    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" event={"ID":"2e469d15-9f4a-4ac3-a422-b8509c7971f8","Type":"ContainerStarted","Data":"00150b309f267aa6eb7a6927a72db4b97217db766d966bd7e073878ffc74eb8b"}
Oct 30 22:01:13.323100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:13.323072    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:13.347576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:13.347527    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podStartSLOduration=2.3475132260000002 podStartE2EDuration="2.347513226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:13.345709927 +0000 UTC m=+1126.434447729" watchObservedRunningTime="2026-10-30 22:01:13.347513226 +0000 UTC m=+1126.436251028"
Oct 30 22:01:17.357224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:17.357189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:19.589279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:19.589250    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.522962 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:01:20.522919    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:01:22.125446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:22.125412    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.323074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:22.323040    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:22.335466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:22.335443    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:23.288970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:23.288942    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:01:26.955371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:26.955330    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.181390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:27.181361    4410 scope.go:122] "RemoveContainer" containerID="b2d5a9196f248f16a353af33013630c0dca43bf9a8fbe67083eeb93586f719c9"
Oct 30 22:01:31.769219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:31.769185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:33.745333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:33.745287    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:33.947198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:33.947148    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:34.328429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:34.328399    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:36.673386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:36.673348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.192109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192077    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:01:41.192482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192453    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:41.192482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192464    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:41.192482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192483    4410 cpu_manager.go:414] "containerMap: removing container" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:41.192590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192488    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63192f50-4032-4e27-9153-7714bbf330dd" containerName="cleanup"
Oct 30 22:01:41.192590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192511    4410 cpu_manager.go:414] "containerMap: removing container" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:41.192590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192516    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f47983-12ba-4ff8-97c0-162443bae1a6" containerName="cleanup"
Oct 30 22:01:41.192679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192597    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efe4dbe-7cea-4edd-8d14-264d2b88ea9b" containerName="cleanup"
Oct 30 22:01:41.192679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.192606    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="249fd36d-6780-4253-9cd9-298061a1b215" containerName="cleanup"
Oct 30 22:01:41.197028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.197011    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.249832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.249798    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.249967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.249848    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.249967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.249940    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.250045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.250006    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.250045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.250039    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.250128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.250114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6mc\" (UniqueName: \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.300526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.300502    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e90e35\"/\"distinct-2-w3-e90e35-kserve-self-signed-certs\""
Oct 30 22:01:41.302988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.302963    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:01:41.314273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.314247    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.351651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351622    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.351830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351670    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.351830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351697    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.351830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351722    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.351830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351751    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.351830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.351829    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6mc\" (UniqueName: \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.352110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.352089    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.352169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.352128    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.352279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.352200    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.354011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.353984    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.354247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.354229    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.409292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.409255    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6mc\" (UniqueName: \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\") pod \"distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") " pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.508674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.508580    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:41.655690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.655655    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.946804 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:01:41.946752    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32cf4881_d14a_4d5d_8455_3d4358009060.slice/crio-7ff6129b077cbd5775dd0a0ed1dc79c5f983c6488e2c22497f88ee59362dc90f WatchSource:0}: Error finding container 7ff6129b077cbd5775dd0a0ed1dc79c5f983c6488e2c22497f88ee59362dc90f: Status 404 returned error can't find the container with id 7ff6129b077cbd5775dd0a0ed1dc79c5f983c6488e2c22497f88ee59362dc90f
Oct 30 22:01:41.954538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:41.954511    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:01:42.378974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:42.378891    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" event={"ID":"32cf4881-d14a-4d5d-8455-3d4358009060","Type":"ContainerStarted","Data":"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"}
Oct 30 22:01:42.378974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:42.378935    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" event={"ID":"32cf4881-d14a-4d5d-8455-3d4358009060","Type":"ContainerStarted","Data":"7ff6129b077cbd5775dd0a0ed1dc79c5f983c6488e2c22497f88ee59362dc90f"}
Oct 30 22:01:42.497578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:42.497519    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podStartSLOduration=1.497502046 podStartE2EDuration="1.497502046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:42.493739823 +0000 UTC m=+1155.582477626" watchObservedRunningTime="2026-10-30 22:01:42.497502046 +0000 UTC m=+1155.586239853"
Oct 30 22:01:42.509563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:42.509536    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:46.565318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:46.565281    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.959105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:49.959047    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.119255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:50.119204    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.524522 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:01:50.524408    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:01:51.445408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:51.445370    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:51.508953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:51.508925    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:51.521339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:51.521314    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:52.442884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:52.442856    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:01:55.657224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:55.657178    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:56.358324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:01:56.358268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:01.241336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:01.241299    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:06.100605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:06.100563    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:06.248448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:06.248413    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:06.440791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:06.440739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:11.155637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:11.155597    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:16.093235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:16.093190    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:19.982351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:19.982310    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:20.567987 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:02:20.567945    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:02:20.879799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:20.879681    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:20.879799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:20.879761    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:21.573075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:21.573040    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:22.732828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:22.732800    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 22:02:22.764069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:22.764032    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:23.599931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.599895    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"]
Oct 30 22:02:23.603660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.603641    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.707316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.707276    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:23.707316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.707283    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:23.707316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.707288    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:23.707594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.707331    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:02:23.710005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.709984    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"]
Oct 30 22:02:23.764441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.764409    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.764870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.764455    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgcp\" (UniqueName: \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.764870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.764540    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.865458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.865370    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.865635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.865492    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.865635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.865523    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgcp\" (UniqueName: \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.866202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.866180    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.867737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.867714    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:23.922551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:23.922523    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgcp\" (UniqueName: \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\") pod \"maas-api-revoke-sub-6a6bccba-hs9z2\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:24.216964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:24.216933    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:24.590068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:24.589981    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"]
Oct 30 22:02:24.592727 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:02:24.592700    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170cf1f4_f583_4ba5_9eac_5d4848c48432.slice/crio-95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9 WatchSource:0}: Error finding container 95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9: Status 404 returned error can't find the container with id 95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9
Oct 30 22:02:24.637209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:24.637185    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2" event={"ID":"170cf1f4-f583-4ba5-9eac-5d4848c48432","Type":"ContainerStarted","Data":"95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9"}
Oct 30 22:02:25.644216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:25.644173    4410 generic.go:356] "Generic (PLEG): container finished" podID="170cf1f4-f583-4ba5-9eac-5d4848c48432" containerID="2308d3a5e7d09b082458f7a80bf0fcb21d48a4620628d8b137a03ce887ada4a4" exitCode=0
Oct 30 22:02:25.644590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:25.644211    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2" event={"ID":"170cf1f4-f583-4ba5-9eac-5d4848c48432","Type":"ContainerDied","Data":"2308d3a5e7d09b082458f7a80bf0fcb21d48a4620628d8b137a03ce887ada4a4"}
Oct 30 22:02:26.574491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.574458    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:26.787483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.787459    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:26.791668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.791650    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\") pod \"170cf1f4-f583-4ba5-9eac-5d4848c48432\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") "
Oct 30 22:02:26.791740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.791724    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\" (UniqueName: \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\") pod \"170cf1f4-f583-4ba5-9eac-5d4848c48432\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") "
Oct 30 22:02:26.791807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.791792    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\") pod \"170cf1f4-f583-4ba5-9eac-5d4848c48432\" (UID: \"170cf1f4-f583-4ba5-9eac-5d4848c48432\") "
Oct 30 22:02:26.792127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.792100    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca" pod "170cf1f4-f583-4ba5-9eac-5d4848c48432" (UID: "170cf1f4-f583-4ba5-9eac-5d4848c48432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:26.792244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.792232    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:26.793628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.793608    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp" pod "170cf1f4-f583-4ba5-9eac-5d4848c48432" (UID: "170cf1f4-f583-4ba5-9eac-5d4848c48432"). InnerVolumeSpecName "kube-api-access-ntgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:26.793799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.793763    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca" pod "170cf1f4-f583-4ba5-9eac-5d4848c48432" (UID: "170cf1f4-f583-4ba5-9eac-5d4848c48432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:26.893519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.893490    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170cf1f4-f583-4ba5-9eac-5d4848c48432-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:26.893519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:26.893516    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgcp\" (UniqueName: \"kubernetes.io/projected/170cf1f4-f583-4ba5-9eac-5d4848c48432-kube-api-access-ntgcp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:27.656818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:27.656769    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"
Oct 30 22:02:27.656818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:27.656804    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2" event={"ID":"170cf1f4-f583-4ba5-9eac-5d4848c48432","Type":"ContainerDied","Data":"95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9"}
Oct 30 22:02:27.657112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:27.656851    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b395d481cbf4536ba6f07b2e8959f52afef5c2dfb3255835bd78cceec2e0c9"
Oct 30 22:02:31.454488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:31.454451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:36.841945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:36.841908    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:37.752433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:37.752382    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" podUID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" containerName="main" containerID="cri-o://0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8" gracePeriod=45
Oct 30 22:02:38.025643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.025618    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 22:02:38.055924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.055900    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.055977    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056050    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056154    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\" (UniqueName: \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056237    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056285    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\") pod \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\" (UID: \"df14351d-2e6a-472d-9f5a-3cac978ad7d0\") "
Oct 30 22:02:38.056516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056282    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.056516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056480    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.056666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056516    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.056739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056722    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.056833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056769    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.056833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.056806    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.058208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.058188    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:38.058648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.058627    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.058648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.058639    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w" pod "df14351d-2e6a-472d-9f5a-3cac978ad7d0" (UID: "df14351d-2e6a-472d-9f5a-3cac978ad7d0"). InnerVolumeSpecName "kube-api-access-gv24w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:38.158299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.158265    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv24w\" (UniqueName: \"kubernetes.io/projected/df14351d-2e6a-472d-9f5a-3cac978ad7d0-kube-api-access-gv24w\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.158299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.158300    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df14351d-2e6a-472d-9f5a-3cac978ad7d0-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.158447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.158315    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df14351d-2e6a-472d-9f5a-3cac978ad7d0-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.732749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.732716    4410 generic.go:356] "Generic (PLEG): container finished" podID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" containerID="0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8" exitCode=0
Oct 30 22:02:38.732947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.732759    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"
Oct 30 22:02:38.732947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.732806    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" event={"ID":"df14351d-2e6a-472d-9f5a-3cac978ad7d0","Type":"ContainerDied","Data":"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"}
Oct 30 22:02:38.732947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.732859    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh" event={"ID":"df14351d-2e6a-472d-9f5a-3cac978ad7d0","Type":"ContainerDied","Data":"7fb83c1ff5de6616c27538059fb0e706c1a8d33dd2661d45511b6f844c5e9474"}
Oct 30 22:02:38.732947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.732886    4410 scope.go:122] "RemoveContainer" containerID="0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"
Oct 30 22:02:38.742008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.741991    4410 scope.go:122] "RemoveContainer" containerID="0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"
Oct 30 22:02:38.742261 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:02:38.742240    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8\": container with ID starting with 0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8 not found: ID does not exist" containerID="0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"
Oct 30 22:02:38.742325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.742269    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8"} err="failed to get container status \"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8\": rpc error: code = NotFound desc = could not find container \"0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8\": container with ID starting with 0448bc146b69279e5085965a68efe4bead33ed9c8d7588f667bf70adffa146e8 not found: ID does not exist"
Oct 30 22:02:38.895438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.895413    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 22:02:38.944619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:38.944590    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-a-9b1cb2d8-kserve-7d7fbb6858-qzkbh"]
Oct 30 22:02:39.019209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:39.019151    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" path="/var/lib/kubelet/pods/df14351d-2e6a-472d-9f5a-3cac978ad7d0/volumes"
Oct 30 22:02:40.012901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.012824    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"]
Oct 30 22:02:40.013256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013239    4410 cpu_manager.go:414] "containerMap: removing container" podUID="170cf1f4-f583-4ba5-9eac-5d4848c48432" containerName="revoke-keys"
Oct 30 22:02:40.013256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013250    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170cf1f4-f583-4ba5-9eac-5d4848c48432" containerName="revoke-keys"
Oct 30 22:02:40.013326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013284    4410 cpu_manager.go:414] "containerMap: removing container" podUID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" containerName="main"
Oct 30 22:02:40.013326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013289    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" containerName="main"
Oct 30 22:02:40.013383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013357    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="170cf1f4-f583-4ba5-9eac-5d4848c48432" containerName="revoke-keys"
Oct 30 22:02:40.013383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.013369    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="df14351d-2e6a-472d-9f5a-3cac978ad7d0" containerName="main"
Oct 30 22:02:40.017690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.017661    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.073194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.073165    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"]
Oct 30 22:02:40.078819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.078794    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.078930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.078889    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.078930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.078919    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgzn\" (UniqueName: \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.119420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.119397    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:40.119554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.119443    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:40.119606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.119549    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:02:40.119606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.119552    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:40.179711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.179684    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.179880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.179742    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.179880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.179766    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgzn\" (UniqueName: \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.180527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.180501    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.181998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.181970    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.237405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.237379    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgzn\" (UniqueName: \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\") pod \"maas-api-revoke-sub-1ba78669-9v6bg\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.331517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.331458    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:40.705362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.705337    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"]
Oct 30 22:02:40.708011 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:02:40.707981    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aedfdd9_c8e8_4a4b_b7e2_90ef97d4eab8.slice/crio-a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8 WatchSource:0}: Error finding container a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8: Status 404 returned error can't find the container with id a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8
Oct 30 22:02:40.744835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:40.744798    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg" event={"ID":"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8","Type":"ContainerStarted","Data":"a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8"}
Oct 30 22:02:41.745757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:41.745718    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:41.751104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:41.751075    4410 generic.go:356] "Generic (PLEG): container finished" podID="9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" containerID="38784dfa9eba20e3f82ef1799369af8fc7a34651b6729ec8938fb1c1e958ee89" exitCode=0
Oct 30 22:02:41.751228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:41.751113    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg" event={"ID":"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8","Type":"ContainerDied","Data":"38784dfa9eba20e3f82ef1799369af8fc7a34651b6729ec8938fb1c1e958ee89"}
Oct 30 22:02:42.892551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:42.892527    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:43.006467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.006439    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\") pod \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") "
Oct 30 22:02:43.006467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.006472    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\") pod \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") "
Oct 30 22:02:43.006695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.006673    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\" (UniqueName: \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\") pod \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\" (UID: \"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8\") "
Oct 30 22:02:43.006858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.006838    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca" pod "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" (UID: "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:43.007061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.007042    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.009099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.009048    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca" pod "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" (UID: "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:43.009188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.009154    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn" pod "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" (UID: "9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8"). InnerVolumeSpecName "kube-api-access-xsgzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:43.108229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.108194    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsgzn\" (UniqueName: \"kubernetes.io/projected/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-kube-api-access-xsgzn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.108229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.108226    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.763334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.763308    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"
Oct 30 22:02:43.763501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.763345    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg" event={"ID":"9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8","Type":"ContainerDied","Data":"a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8"}
Oct 30 22:02:43.763501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:43.763391    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1966087e9973d681d143bb50a90e6c96a7ea6e76a6ee0c702458a17f541c4e8"
Oct 30 22:02:46.555072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:46.555036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:47.677812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.677764    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:02:47.678211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.678197    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" containerName="revoke-keys"
Oct 30 22:02:47.678254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.678212    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" containerName="revoke-keys"
Oct 30 22:02:47.678310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.678300    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" containerName="revoke-keys"
Oct 30 22:02:47.680943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.680926    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755041    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755124    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755245    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpg2\" (UniqueName: \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755349    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755388    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.755516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.755500    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.781512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.781486    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d2a3e0\"/\"distinct-2-w4-d2a3e0-kserve-self-signed-certs\""
Oct 30 22:02:47.784940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.784908    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:02:47.786023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.786000    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:47.856190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856164    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856212    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856259    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpg2\" (UniqueName: \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856317    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856355    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856499    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856614    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856707    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.856956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.856931    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.858931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.858910    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.861314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.861269    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.913429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.913388    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpg2\" (UniqueName: \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\") pod \"distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") " pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:47.992832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:47.992703    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:48.393172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:48.393146    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:02:48.394707 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:02:48.394674    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e4c12a7_6a75_4c34_ad98_342b571977a0.slice/crio-588400eb8cf830512613667ecadac28fa6f033f1eb0a7802454c8cec42a945ef WatchSource:0}: Error finding container 588400eb8cf830512613667ecadac28fa6f033f1eb0a7802454c8cec42a945ef: Status 404 returned error can't find the container with id 588400eb8cf830512613667ecadac28fa6f033f1eb0a7802454c8cec42a945ef
Oct 30 22:02:48.798528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:48.798432    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" event={"ID":"5e4c12a7-6a75-4c34-ad98-342b571977a0","Type":"ContainerStarted","Data":"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"}
Oct 30 22:02:48.798528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:48.798482    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" event={"ID":"5e4c12a7-6a75-4c34-ad98-342b571977a0","Type":"ContainerStarted","Data":"588400eb8cf830512613667ecadac28fa6f033f1eb0a7802454c8cec42a945ef"}
Oct 30 22:02:48.913001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:48.912953    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podStartSLOduration=1.9129405560000001 podStartE2EDuration="1.912940556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:48.912733625 +0000 UTC m=+1222.001471441" watchObservedRunningTime="2026-10-30 22:02:48.912940556 +0000 UTC m=+1222.001678359"
Oct 30 22:02:48.993387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:48.993351    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:50.522289 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:02:50.522254    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:02:51.054574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:51.054544    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"]
Oct 30 22:02:51.103468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:51.103439    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52aefa85-87dps"]
Oct 30 22:02:51.444194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:51.444151    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:53.018549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:53.018514    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee06c6bb-9155-4737-8224-b2ce32c91416" path="/var/lib/kubelet/pods/ee06c6bb-9155-4737-8224-b2ce32c91416/volumes"
Oct 30 22:02:56.022048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.022012    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"]
Oct 30 22:02:56.025748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.025731    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.128704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.128678    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:56.128890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.128757    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:02:56.128890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.128801    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:56.128890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.128802    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:56.131500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.131481    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"]
Oct 30 22:02:56.142668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.142643    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rxm\" (UniqueName: \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.142855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.142834    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.142933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.142903    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.244058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.244032    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rxm\" (UniqueName: \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.244200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.244084    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.244262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.244248    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.244652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.244633    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.246540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.246523    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.301329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.301273    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rxm\" (UniqueName: \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\") pod \"maas-api-revoke-sub-2984ae73-jj5pb\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.339227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.339202    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:56.401973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.401846    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:56.550823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.550792    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"]
Oct 30 22:02:56.554462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.554417    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.612643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.612615    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"]
Oct 30 22:02:56.648987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.648963    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.649096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.649068    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djk87\" (UniqueName: \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.649145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.649125    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.724989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.724964    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"]
Oct 30 22:02:56.726474 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:02:56.726452    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4ba892_6d80_4d0a_91e9_862dca504d8a.slice/crio-2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545 WatchSource:0}: Error finding container 2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545: Status 404 returned error can't find the container with id 2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545
Oct 30 22:02:56.750046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.750023    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.750149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.750085    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djk87\" (UniqueName: \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.750224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.750198    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.750740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.750722    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.752130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.752114    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.806544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.806494    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djk87\" (UniqueName: \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\") pod \"maas-api-revoke-sub-f9b36e14-jmm7t\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:56.846091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.846069    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb" event={"ID":"7a4ba892-6d80-4d0a-91e9-862dca504d8a","Type":"ContainerStarted","Data":"2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545"}
Oct 30 22:02:56.867495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:56.867477    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:02:57.239144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.239115    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"]
Oct 30 22:02:57.240784 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:02:57.240747    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d73aa03_f578_42b3_bed3_9ff90ee16d1a.slice/crio-19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b WatchSource:0}: Error finding container 19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b: Status 404 returned error can't find the container with id 19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b
Oct 30 22:02:57.853536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.853443    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t" event={"ID":"6d73aa03-f578-42b3-bed3-9ff90ee16d1a","Type":"ContainerStarted","Data":"808a6640c25bb0dd82834e9f14b989d2c3f1f178c07fce33dc1eaf1c1671df2a"}
Oct 30 22:02:57.853536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.853511    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t" event={"ID":"6d73aa03-f578-42b3-bed3-9ff90ee16d1a","Type":"ContainerStarted","Data":"19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b"}
Oct 30 22:02:57.855158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.855127    4410 generic.go:356] "Generic (PLEG): container finished" podID="7a4ba892-6d80-4d0a-91e9-862dca504d8a" containerID="e5769e94d2f97933ea27d398158ff6dc06310427b8d5362c5d87caa81867ea16" exitCode=0
Oct 30 22:02:57.855284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.855201    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb" event={"ID":"7a4ba892-6d80-4d0a-91e9-862dca504d8a","Type":"ContainerDied","Data":"e5769e94d2f97933ea27d398158ff6dc06310427b8d5362c5d87caa81867ea16"}
Oct 30 22:02:57.967255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.967210    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t" podStartSLOduration=1.9671972 podStartE2EDuration="1.9671972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:57.966649313 +0000 UTC m=+1231.055387117" watchObservedRunningTime="2026-10-30 22:02:57.9671972 +0000 UTC m=+1231.055935004"
Oct 30 22:02:57.993032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:57.993006    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:58.006037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:58.006014    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:58.863100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:58.863049    4410 generic.go:356] "Generic (PLEG): container finished" podID="6d73aa03-f578-42b3-bed3-9ff90ee16d1a" containerID="808a6640c25bb0dd82834e9f14b989d2c3f1f178c07fce33dc1eaf1c1671df2a" exitCode=0
Oct 30 22:02:58.863576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:58.863140    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t" event={"ID":"6d73aa03-f578-42b3-bed3-9ff90ee16d1a","Type":"ContainerDied","Data":"808a6640c25bb0dd82834e9f14b989d2c3f1f178c07fce33dc1eaf1c1671df2a"}
Oct 30 22:02:58.874815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:58.874767    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:02:59.011254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.011226    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:59.173819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.173784    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\" (UniqueName: \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\") pod \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") "
Oct 30 22:02:59.173984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.173896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\") pod \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") "
Oct 30 22:02:59.173984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.173942    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\") pod \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\" (UID: \"7a4ba892-6d80-4d0a-91e9-862dca504d8a\") "
Oct 30 22:02:59.174363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.174339    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca" pod "7a4ba892-6d80-4d0a-91e9-862dca504d8a" (UID: "7a4ba892-6d80-4d0a-91e9-862dca504d8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:59.175940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.175914    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca" pod "7a4ba892-6d80-4d0a-91e9-862dca504d8a" (UID: "7a4ba892-6d80-4d0a-91e9-862dca504d8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:59.176032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.175952    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm" pod "7a4ba892-6d80-4d0a-91e9-862dca504d8a" (UID: "7a4ba892-6d80-4d0a-91e9-862dca504d8a"). InnerVolumeSpecName "kube-api-access-n9rxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:59.275514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.275484    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rxm\" (UniqueName: \"kubernetes.io/projected/7a4ba892-6d80-4d0a-91e9-862dca504d8a-kube-api-access-n9rxm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:59.275514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.275514    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:59.275705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.275528    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4ba892-6d80-4d0a-91e9-862dca504d8a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:59.871706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.871674    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"
Oct 30 22:02:59.872145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.871705    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb" event={"ID":"7a4ba892-6d80-4d0a-91e9-862dca504d8a","Type":"ContainerDied","Data":"2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545"}
Oct 30 22:02:59.872145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:02:59.871759    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f14f43fb343cb3c101645cc0c305270927812ae3fc4ea34c7ecaf20371c9545"
Oct 30 22:03:00.011046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.011022    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:03:00.185034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.185009    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\" (UniqueName: \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\") pod \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") "
Oct 30 22:03:00.185231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.185101    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\") pod \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") "
Oct 30 22:03:00.185231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.185190    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\") pod \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\" (UID: \"6d73aa03-f578-42b3-bed3-9ff90ee16d1a\") "
Oct 30 22:03:00.185502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.185477    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca" pod "6d73aa03-f578-42b3-bed3-9ff90ee16d1a" (UID: "6d73aa03-f578-42b3-bed3-9ff90ee16d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:00.187047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.187027    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca" pod "6d73aa03-f578-42b3-bed3-9ff90ee16d1a" (UID: "6d73aa03-f578-42b3-bed3-9ff90ee16d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:00.187133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.187111    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87" pod "6d73aa03-f578-42b3-bed3-9ff90ee16d1a" (UID: "6d73aa03-f578-42b3-bed3-9ff90ee16d1a"). InnerVolumeSpecName "kube-api-access-djk87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:00.286327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.286301    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.286327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.286327    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djk87\" (UniqueName: \"kubernetes.io/projected/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-kube-api-access-djk87\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.286480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.286343    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d73aa03-f578-42b3-bed3-9ff90ee16d1a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.878091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.878063    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"
Oct 30 22:03:00.878475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.878065    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t" event={"ID":"6d73aa03-f578-42b3-bed3-9ff90ee16d1a","Type":"ContainerDied","Data":"19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b"}
Oct 30 22:03:00.878475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:00.878196    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19cff6d1231da2805b146b79d772f5f3738c6c663a389be9d7dd0cd316172e9b"
Oct 30 22:03:01.297809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:01.297760    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:06.270357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.270311    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:06.767964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.767929    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:03:06.768362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.768350    4410 cpu_manager.go:414] "containerMap: removing container" podUID="7a4ba892-6d80-4d0a-91e9-862dca504d8a" containerName="revoke-keys"
Oct 30 22:03:06.768402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.768364    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4ba892-6d80-4d0a-91e9-862dca504d8a" containerName="revoke-keys"
Oct 30 22:03:06.768402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.768372    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6d73aa03-f578-42b3-bed3-9ff90ee16d1a" containerName="revoke-keys"
Oct 30 22:03:06.768402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.768377    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d73aa03-f578-42b3-bed3-9ff90ee16d1a" containerName="revoke-keys"
Oct 30 22:03:06.771874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.769091    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4ba892-6d80-4d0a-91e9-862dca504d8a" containerName="revoke-keys"
Oct 30 22:03:06.771874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.769125    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d73aa03-f578-42b3-bed3-9ff90ee16d1a" containerName="revoke-keys"
Oct 30 22:03:06.775823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.774150    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.827120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.827092    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1c3eef-openshift-default-dockercfg-hgpgr\""
Oct 30 22:03:06.852797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.852749    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.852922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.852814    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjp92\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.852922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.852883    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.852995    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.853059    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.853076    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.853096    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.853189    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.853252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.853231    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.879429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.879400    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:03:06.954051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954019    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954110    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954134    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954162    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954201    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954260    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954296    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954343    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjp92\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954370    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954558    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954691    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954912    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.954937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.954924    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.955227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.955205    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.956491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.956458    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:06.956947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:06.956924    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:07.011898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.011860    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:07.012149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.012105    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjp92\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:07.088022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.087953    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:07.298091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.298061    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"]
Oct 30 22:03:07.302568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.302549    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.357154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.357101    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:07.357374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.357356    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:07.357514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.357375    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:07.357986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.357933    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.358077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.358033    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.358150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.358120    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjd2\" (UniqueName: \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.358150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.358133    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:07.360931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.360910    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"]
Oct 30 22:03:07.459141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.459110    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.459141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.459143    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.459315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.459185    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjd2\" (UniqueName: \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.459878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.459852    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.461592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.461572    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.513568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.513542    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:03:07.515668 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:07.515638    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac26be9_11e5_4d80_a7bf_37e37a6034b9.slice/crio-f2492c8aba6b3b03882837eb5fe5e27a4c64ee8ca8ca53ee4f9ed278d238f362 WatchSource:0}: Error finding container f2492c8aba6b3b03882837eb5fe5e27a4c64ee8ca8ca53ee4f9ed278d238f362: Status 404 returned error can't find the container with id f2492c8aba6b3b03882837eb5fe5e27a4c64ee8ca8ca53ee4f9ed278d238f362
Oct 30 22:03:07.516613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.516593    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjd2\" (UniqueName: \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\") pod \"maas-api-revoke-sub-dffa1af5-98bxh\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.518032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.518002    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:07.518119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.518060    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:07.518119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.518100    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:07.616510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.616438    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:07.938162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.938104    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" event={"ID":"3ac26be9-11e5-4d80-a7bf-37e37a6034b9","Type":"ContainerStarted","Data":"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"}
Oct 30 22:03:07.938357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.938168    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" event={"ID":"3ac26be9-11e5-4d80-a7bf-37e37a6034b9","Type":"ContainerStarted","Data":"f2492c8aba6b3b03882837eb5fe5e27a4c64ee8ca8ca53ee4f9ed278d238f362"}
Oct 30 22:03:07.991166 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:07.991137    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce88a89b_482e_4230_8861_fceda7ac3214.slice/crio-34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90 WatchSource:0}: Error finding container 34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90: Status 404 returned error can't find the container with id 34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90
Oct 30 22:03:07.991372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:07.991349    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"]
Oct 30 22:03:08.064815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.064237    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podStartSLOduration=2.064216008 podStartE2EDuration="2.064216008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:08.059951252 +0000 UTC m=+1241.148689056" watchObservedRunningTime="2026-10-30 22:03:08.064216008 +0000 UTC m=+1241.152953813"
Oct 30 22:03:08.088804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.088748    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:08.090454 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.090423    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:08.090553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.090485    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:08.965820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.965356    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh" event={"ID":"ce88a89b-482e-4230-8861-fceda7ac3214","Type":"ContainerStarted","Data":"8a7c18078b18e129557724a5e809438696760c43a53d081a53298260ea39feb3"}
Oct 30 22:03:08.965820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:08.965417    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh" event={"ID":"ce88a89b-482e-4230-8861-fceda7ac3214","Type":"ContainerStarted","Data":"34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90"}
Oct 30 22:03:09.070798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.069827    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"]
Oct 30 22:03:09.072494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.072466    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22858176-stzjw"]
Oct 30 22:03:09.088466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.088438    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:09.088581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.088489    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:09.974638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.974604    4410 generic.go:356] "Generic (PLEG): container finished" podID="ce88a89b-482e-4230-8861-fceda7ac3214" containerID="8a7c18078b18e129557724a5e809438696760c43a53d081a53298260ea39feb3" exitCode=0
Oct 30 22:03:09.975160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:09.974636    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh" event={"ID":"ce88a89b-482e-4230-8861-fceda7ac3214","Type":"ContainerDied","Data":"8a7c18078b18e129557724a5e809438696760c43a53d081a53298260ea39feb3"}
Oct 30 22:03:10.088680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.088646    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:10.088862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.088697    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:10.130837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.130811    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:10.191483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.191439    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\") pod \"ce88a89b-482e-4230-8861-fceda7ac3214\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") "
Oct 30 22:03:10.191665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.191551    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\" (UniqueName: \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\") pod \"ce88a89b-482e-4230-8861-fceda7ac3214\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") "
Oct 30 22:03:10.191665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.191601    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\") pod \"ce88a89b-482e-4230-8861-fceda7ac3214\" (UID: \"ce88a89b-482e-4230-8861-fceda7ac3214\") "
Oct 30 22:03:10.191994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.191971    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca" pod "ce88a89b-482e-4230-8861-fceda7ac3214" (UID: "ce88a89b-482e-4230-8861-fceda7ac3214"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:10.193406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.193382    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca" pod "ce88a89b-482e-4230-8861-fceda7ac3214" (UID: "ce88a89b-482e-4230-8861-fceda7ac3214"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:10.193529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.193510    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2" pod "ce88a89b-482e-4230-8861-fceda7ac3214" (UID: "ce88a89b-482e-4230-8861-fceda7ac3214"). InnerVolumeSpecName "kube-api-access-kxjd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:10.292570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.292489    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.292570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.292516    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjd2\" (UniqueName: \"kubernetes.io/projected/ce88a89b-482e-4230-8861-fceda7ac3214-kube-api-access-kxjd2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.292570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.292526    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88a89b-482e-4230-8861-fceda7ac3214-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.981077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.981036    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh" event={"ID":"ce88a89b-482e-4230-8861-fceda7ac3214","Type":"ContainerDied","Data":"34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90"}
Oct 30 22:03:10.981570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.981081    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"
Oct 30 22:03:10.981570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:10.981126    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34fbc84dfd6f648957745e63f595059f41d89d14d177ed74c40a4eb93e0e9e90"
Oct 30 22:03:11.018921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:11.018888    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18069dd1-417e-4eb6-a1a0-570913d37ce8" path="/var/lib/kubelet/pods/18069dd1-417e-4eb6-a1a0-570913d37ce8/volumes"
Oct 30 22:03:11.088343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:11.088308    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:11.088509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:11.088372    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:11.172795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:11.172013    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:12.088517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:12.088481    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:12.088944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:12.088545    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:13.088663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.088631    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:13.089062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.088690    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:13.836299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.836265    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"]
Oct 30 22:03:13.836752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.836735    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ce88a89b-482e-4230-8861-fceda7ac3214" containerName="revoke-keys"
Oct 30 22:03:13.836860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.836753    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce88a89b-482e-4230-8861-fceda7ac3214" containerName="revoke-keys"
Oct 30 22:03:13.836899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.836876    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce88a89b-482e-4230-8861-fceda7ac3214" containerName="revoke-keys"
Oct 30 22:03:13.840451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.840432    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:13.893281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.893256    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:13.929077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.929049    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:13.929225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.929085    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrq5r\" (UniqueName: \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:13.929225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.929118    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:13.942458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.942429    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:13.942458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.942442    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:13.942683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.942575    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:13.946824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:13.946796    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"]
Oct 30 22:03:14.030544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.030510    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.030544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.030545    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrq5r\" (UniqueName: \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.030802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.030579    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.031272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.031250    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.032889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.032864    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.087261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.087189    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrq5r\" (UniqueName: \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\") pod \"maas-api-revoke-sub-53af5867-8g4w9\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.089082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.089054    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:03:14.089494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.089106    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 22:03:14.157357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.157329    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:14.534320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.534291    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"]
Oct 30 22:03:14.536855 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:14.536821    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308a692_491a_444b_a52e_9c4755d6c9c1.slice/crio-49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb WatchSource:0}: Error finding container 49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb: Status 404 returned error can't find the container with id 49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb
Oct 30 22:03:14.629803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:14.629739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:15.007623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.007577    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9" event={"ID":"f308a692-491a-444b-a52e-9c4755d6c9c1","Type":"ContainerStarted","Data":"e1edd68006874026be3be4cffb6d8d502a082e0d91cab78798ee40f66b207149"}
Oct 30 22:03:15.007811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.007650    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9" event={"ID":"f308a692-491a-444b-a52e-9c4755d6c9c1","Type":"ContainerStarted","Data":"49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb"}
Oct 30 22:03:15.093096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.093069    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:15.093540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.093439    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:15.094131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.094111    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:03:15.133040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:15.132951    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9" podStartSLOduration=2.132936084 podStartE2EDuration="2.132936084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.132799082 +0000 UTC m=+1248.221536886" watchObservedRunningTime="2026-10-30 22:03:15.132936084 +0000 UTC m=+1248.221673883"
Oct 30 22:03:16.014670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.014632    4410 generic.go:356] "Generic (PLEG): container finished" podID="f308a692-491a-444b-a52e-9c4755d6c9c1" containerID="e1edd68006874026be3be4cffb6d8d502a082e0d91cab78798ee40f66b207149" exitCode=0
Oct 30 22:03:16.014901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.014717    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9" event={"ID":"f308a692-491a-444b-a52e-9c4755d6c9c1","Type":"ContainerDied","Data":"e1edd68006874026be3be4cffb6d8d502a082e0d91cab78798ee40f66b207149"}
Oct 30 22:03:16.070585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.070545    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:16.765455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.765420    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:03:16.769083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.769063    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.821122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.821091    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d2a3e0-openshift-default-dockercfg-t4vpk\""
Oct 30 22:03:16.861110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861070    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861131    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861167    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861197    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861229    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tjq\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861270    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861353    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861381    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.861582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.861477    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.875405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.874948    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:03:16.962730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962701    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.962952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962738    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.962952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962760    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tjq\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.962952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962808    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.962952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962875    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.962952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962894    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.962983    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963035    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963075    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963201    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963366    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963458    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963546    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.963594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.963582    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.965237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.965219    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:16.965461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:16.965442    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:17.019345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.019275    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:17.019345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.019287    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tjq\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:17.084194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.084145    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:17.170011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.168848    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:17.266013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.265990    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\") pod \"f308a692-491a-444b-a52e-9c4755d6c9c1\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") "
Oct 30 22:03:17.266154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.266120    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\" (UniqueName: \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\") pod \"f308a692-491a-444b-a52e-9c4755d6c9c1\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") "
Oct 30 22:03:17.266237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.266191    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\") pod \"f308a692-491a-444b-a52e-9c4755d6c9c1\" (UID: \"f308a692-491a-444b-a52e-9c4755d6c9c1\") "
Oct 30 22:03:17.266539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.266507    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca" pod "f308a692-491a-444b-a52e-9c4755d6c9c1" (UID: "f308a692-491a-444b-a52e-9c4755d6c9c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:17.268163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.268137    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca" pod "f308a692-491a-444b-a52e-9c4755d6c9c1" (UID: "f308a692-491a-444b-a52e-9c4755d6c9c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:17.268163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.268140    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r" pod "f308a692-491a-444b-a52e-9c4755d6c9c1" (UID: "f308a692-491a-444b-a52e-9c4755d6c9c1"). InnerVolumeSpecName "kube-api-access-nrq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:17.367759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.367678    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.367759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.367710    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f308a692-491a-444b-a52e-9c4755d6c9c1-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.367759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.367726    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrq5r\" (UniqueName: \"kubernetes.io/projected/f308a692-491a-444b-a52e-9c4755d6c9c1-kube-api-access-nrq5r\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.523275 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.523242    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:03:17.523997 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:17.523966    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780eaa75_630a_4d26_8b15_91e18de95b26.slice/crio-029156284e154a63f390f9cd849273efcf197a8deb79d13cfacee2b9fb978858 WatchSource:0}: Error finding container 029156284e154a63f390f9cd849273efcf197a8deb79d13cfacee2b9fb978858: Status 404 returned error can't find the container with id 029156284e154a63f390f9cd849273efcf197a8deb79d13cfacee2b9fb978858
Oct 30 22:03:17.526591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.526551    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:17.526684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.526637    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:17.526723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:17.526696    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:18.031795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.031515    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"
Oct 30 22:03:18.031795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.031530    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9" event={"ID":"f308a692-491a-444b-a52e-9c4755d6c9c1","Type":"ContainerDied","Data":"49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb"}
Oct 30 22:03:18.031795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.031603    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49625a5da0c4379d3d0a9beaa5d9a68c3f6f026650573be32132d64407383bdb"
Oct 30 22:03:18.034372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.034347    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" event={"ID":"780eaa75-630a-4d26-8b15-91e18de95b26","Type":"ContainerStarted","Data":"145ce53da28465145b0de5e546d52a338c541e3180c3fca003cb8fab543c7900"}
Oct 30 22:03:18.034516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.034385    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" event={"ID":"780eaa75-630a-4d26-8b15-91e18de95b26","Type":"ContainerStarted","Data":"029156284e154a63f390f9cd849273efcf197a8deb79d13cfacee2b9fb978858"}
Oct 30 22:03:18.085114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.085084    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:18.086872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.086843    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:18.087016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.086892    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:18.153005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:18.152945    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podStartSLOduration=2.152926634 podStartE2EDuration="2.152926634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:18.152836057 +0000 UTC m=+1251.241573861" watchObservedRunningTime="2026-10-30 22:03:18.152926634 +0000 UTC m=+1251.241664438"
Oct 30 22:03:19.084865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:19.084825    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:19.085328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:19.084877    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:20.085061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:20.085022    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:20.085462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:20.085076    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:20.329205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:20.329166    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:20.521904 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:03:20.521864    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:03:20.997455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:20.997418    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:21.084852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.084814    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:21.085031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.084878    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:21.901515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.901483    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:03:21.901982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.901966    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f308a692-491a-444b-a52e-9c4755d6c9c1" containerName="revoke-keys"
Oct 30 22:03:21.902047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.901984    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f308a692-491a-444b-a52e-9c4755d6c9c1" containerName="revoke-keys"
Oct 30 22:03:21.902089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.902078    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f308a692-491a-444b-a52e-9c4755d6c9c1" containerName="revoke-keys"
Oct 30 22:03:21.905557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:21.905540    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018165    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018329    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018437    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018468    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ghg\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018595    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018666    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018791    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018844    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.018923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.018901    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.020041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.020022    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:03:22.084924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.084890    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:22.085082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.084942    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:22.119724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119686    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.119874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119754    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.119874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119821    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.119874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119861    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119889    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119950    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.119998    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120024    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ghg\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120075    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120089    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120249    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120578    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.120703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.120685    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.121094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.121071    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.122219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.122200    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.122509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.122488    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.176656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.176593    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.177273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.177246    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ghg\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.217756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.217731    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:22.408391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.408355    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"]
Oct 30 22:03:22.413153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.413133    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.471898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.470112    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:22.471898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.470241    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:22.471898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.470286    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:22.471898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.470120    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:22.474797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.474656    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"]
Oct 30 22:03:22.524059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.524032    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.524212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.524086    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.524262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.524235    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rzn\" (UniqueName: \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.625059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.625024    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rzn\" (UniqueName: \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.625226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.625087    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.625226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.625132    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.625744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.625723    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.627450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.627432    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.675257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.675231    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:03:22.675765 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:22.675740    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d65ac8b_c556_4dc0_bdb4_317d214bc465.slice/crio-154e0f77789e0ea26d1ba27d1fc0e6ee8e943c1c283fc69e2ccf143d73637aad WatchSource:0}: Error finding container 154e0f77789e0ea26d1ba27d1fc0e6ee8e943c1c283fc69e2ccf143d73637aad: Status 404 returned error can't find the container with id 154e0f77789e0ea26d1ba27d1fc0e6ee8e943c1c283fc69e2ccf143d73637aad
Oct 30 22:03:22.678094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.678063    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:22.678169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.678129    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:22.678169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.678156    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:22.681556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.681532    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rzn\" (UniqueName: \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\") pod \"maas-api-revoke-sub-626739cf-bfgpr\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:22.726744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:22.726724    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:23.085551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.085472    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:23.085551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.085528    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:23.090564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.090531    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" event={"ID":"1d65ac8b-c556-4dc0-bdb4-317d214bc465","Type":"ContainerStarted","Data":"196d78c663104d68b6c22a15f85c276f759516e713006eb0248f81906c35702d"}
Oct 30 22:03:23.090714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.090588    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" event={"ID":"1d65ac8b-c556-4dc0-bdb4-317d214bc465","Type":"ContainerStarted","Data":"154e0f77789e0ea26d1ba27d1fc0e6ee8e943c1c283fc69e2ccf143d73637aad"}
Oct 30 22:03:23.135127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.135101    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"]
Oct 30 22:03:23.136768 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:23.136741    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145f98ec_bd2a_48bf_8275_05f1a33bb4d0.slice/crio-d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6 WatchSource:0}: Error finding container d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6: Status 404 returned error can't find the container with id d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6
Oct 30 22:03:23.214324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.214274    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podStartSLOduration=2.214259353 podStartE2EDuration="2.214259353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:23.211492706 +0000 UTC m=+1256.300230511" watchObservedRunningTime="2026-10-30 22:03:23.214259353 +0000 UTC m=+1256.302997155"
Oct 30 22:03:23.218836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.218811    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:23.220498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.220465    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:23.220581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:23.220518    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:24.084613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.084572    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:03:24.084844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.084629    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 22:03:24.100605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.100489    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr" event={"ID":"145f98ec-bd2a-48bf-8275-05f1a33bb4d0","Type":"ContainerStarted","Data":"85e842658769d3f5f2b2ca5efc0931e997f7ade68e877b2b489ac768a1258729"}
Oct 30 22:03:24.100605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.100564    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr" event={"ID":"145f98ec-bd2a-48bf-8275-05f1a33bb4d0","Type":"ContainerStarted","Data":"d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6"}
Oct 30 22:03:24.221803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.221656    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:24.221803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.221724    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:24.237796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:24.237205    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr" podStartSLOduration=2.237188754 podStartE2EDuration="2.237188754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:24.237027771 +0000 UTC m=+1257.325765598" watchObservedRunningTime="2026-10-30 22:03:24.237188754 +0000 UTC m=+1257.325926560"
Oct 30 22:03:25.112797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.112007    4410 generic.go:356] "Generic (PLEG): container finished" podID="145f98ec-bd2a-48bf-8275-05f1a33bb4d0" containerID="85e842658769d3f5f2b2ca5efc0931e997f7ade68e877b2b489ac768a1258729" exitCode=0
Oct 30 22:03:25.113263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.113143    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr" event={"ID":"145f98ec-bd2a-48bf-8275-05f1a33bb4d0","Type":"ContainerDied","Data":"85e842658769d3f5f2b2ca5efc0931e997f7ade68e877b2b489ac768a1258729"}
Oct 30 22:03:25.218795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.218308    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:25.218795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.218367    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:25.330988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.330956    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:25.331983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.331809    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:25.333553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.333528    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:03:25.369844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.367162    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:03:25.384073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.384039    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.414871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.414834    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:03:25.489204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489155    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489316    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489363    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489471    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489517    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489562    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489621    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489700    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.489757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.489746    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcwp\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.595433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.595389    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.595727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.595711    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.595865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.595850    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596005    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596124    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596231    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596235    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596491    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596650    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.596803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.596786    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcwp\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.597477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.597454    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.597843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.597825    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.598496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.598477    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.599390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.599367    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.600091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.600072    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.601211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.601192    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.669334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.669299    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.670138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.670113    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcwp\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:25.732829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:25.732767    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:26.049798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.046593    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"]
Oct 30 22:03:26.070844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.070766    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932d0ecb-zclsg"]
Oct 30 22:03:26.219278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.219230    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:26.219702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.219354    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:26.291463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.291433    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:26.295478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.295457    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:03:26.297444 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:26.297418    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cb44b3_e782_4c60_8323_4a7ee0cc7cb9.slice/crio-2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0 WatchSource:0}: Error finding container 2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0: Status 404 returned error can't find the container with id 2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0
Oct 30 22:03:26.300049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.300020    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:26.300108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.300082    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:26.300140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.300125    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:26.408604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.408560    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\") pod \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") "
Oct 30 22:03:26.408766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.408615    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\") pod \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") "
Oct 30 22:03:26.408766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.408725    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\" (UniqueName: \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\") pod \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\" (UID: \"145f98ec-bd2a-48bf-8275-05f1a33bb4d0\") "
Oct 30 22:03:26.410115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.409818    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca" pod "145f98ec-bd2a-48bf-8275-05f1a33bb4d0" (UID: "145f98ec-bd2a-48bf-8275-05f1a33bb4d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:26.412003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.411968    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn" pod "145f98ec-bd2a-48bf-8275-05f1a33bb4d0" (UID: "145f98ec-bd2a-48bf-8275-05f1a33bb4d0"). InnerVolumeSpecName "kube-api-access-94rzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:26.414885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.414860    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca" pod "145f98ec-bd2a-48bf-8275-05f1a33bb4d0" (UID: "145f98ec-bd2a-48bf-8275-05f1a33bb4d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:26.510667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.510594    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.510667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.510620    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.510667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.510631    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94rzn\" (UniqueName: \"kubernetes.io/projected/145f98ec-bd2a-48bf-8275-05f1a33bb4d0-kube-api-access-94rzn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.738841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.738347    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:26.919579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:26.919528    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:27.019749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.019715    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947838c7-f014-4695-b9b8-04e78bfc5a21" path="/var/lib/kubelet/pods/947838c7-f014-4695-b9b8-04e78bfc5a21/volumes"
Oct 30 22:03:27.071573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.071495    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"]
Oct 30 22:03:27.075292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.075248    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:27.075608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.075588    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1226f44-rvt4d"]
Oct 30 22:03:27.124005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.123980    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"
Oct 30 22:03:27.124130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.123979    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr" event={"ID":"145f98ec-bd2a-48bf-8275-05f1a33bb4d0","Type":"ContainerDied","Data":"d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6"}
Oct 30 22:03:27.124130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.124102    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22428910537c7ad8ce4eb0543fcee8b537162af1e8935a4b0f8f2de87bf91b6"
Oct 30 22:03:27.125710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.125685    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" event={"ID":"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9","Type":"ContainerStarted","Data":"7d2b1f3017518e83ebd57279320e9701d40466d74779f69b21aca19a0a441166"}
Oct 30 22:03:27.125850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.125728    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" event={"ID":"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9","Type":"ContainerStarted","Data":"2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0"}
Oct 30 22:03:27.219133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.218161    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:27.219133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.218217    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:27.274384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.274330    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podStartSLOduration=2.274315806 podStartE2EDuration="2.274315806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:27.259081664 +0000 UTC m=+1260.347819468" watchObservedRunningTime="2026-10-30 22:03:27.274315806 +0000 UTC m=+1260.363053608"
Oct 30 22:03:27.733873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.733836    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:27.735257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.735235    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:27.735330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:27.735285    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:28.218800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.218739    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"]
Oct 30 22:03:28.219002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.218905    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.219002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.218949    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:28.219440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.219420    4410 cpu_manager.go:414] "containerMap: removing container" podUID="145f98ec-bd2a-48bf-8275-05f1a33bb4d0" containerName="revoke-keys"
Oct 30 22:03:28.219440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.219440    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145f98ec-bd2a-48bf-8275-05f1a33bb4d0" containerName="revoke-keys"
Oct 30 22:03:28.219946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.219615    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="145f98ec-bd2a-48bf-8275-05f1a33bb4d0" containerName="revoke-keys"
Oct 30 22:03:28.224444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.224425    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.293797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.293091    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:28.294796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.294101    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:28.298190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.298160    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"]
Oct 30 22:03:28.322585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.322559    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:28.322698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.322631    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:28.338677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.338649    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.338759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.338723    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.338759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.338751    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzvl\" (UniqueName: \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.439514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.439481    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.439656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.439548    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.439656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.439577    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzvl\" (UniqueName: \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.440317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.440294    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.441866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.441845    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.500471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.500382    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzvl\" (UniqueName: \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\") pod \"maas-api-revoke-sub-fa030e17-2hh2w\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.538193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.538157    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:28.736062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.735551    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.736062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:28.735609    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:29.002176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.002148    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"]
Oct 30 22:03:29.004123 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:29.004092    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213f219c_be97_4ec7_8edb_db3a9a4f3bc0.slice/crio-0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64 WatchSource:0}: Error finding container 0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64: Status 404 returned error can't find the container with id 0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64
Oct 30 22:03:29.020880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.020855    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45940d30-4d56-4c45-90be-14c4b5006935" path="/var/lib/kubelet/pods/45940d30-4d56-4c45-90be-14c4b5006935/volumes"
Oct 30 22:03:29.139638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.139602    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w" event={"ID":"213f219c-be97-4ec7-8edb-db3a9a4f3bc0","Type":"ContainerStarted","Data":"0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64"}
Oct 30 22:03:29.218665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.218638    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.218837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.218686    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:03:29.733718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.733679    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.734130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:29.733744    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:30.177714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.177673    4410 generic.go:356] "Generic (PLEG): container finished" podID="213f219c-be97-4ec7-8edb-db3a9a4f3bc0" containerID="a452d293261df98a667ba8d68bc801dcb7e5df4bf78349236056ebbb33969342" exitCode=0
Oct 30 22:03:30.177926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.177745    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w" event={"ID":"213f219c-be97-4ec7-8edb-db3a9a4f3bc0","Type":"ContainerDied","Data":"a452d293261df98a667ba8d68bc801dcb7e5df4bf78349236056ebbb33969342"}
Oct 30 22:03:30.498919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.493600    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:30.498919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.494095    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:30.498919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.494945    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:03:30.733430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.733385    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:30.733687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:30.733440    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:31.435515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.435492    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:31.504183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.504093    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\" (UniqueName: \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\") pod \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") "
Oct 30 22:03:31.504183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.504150    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\") pod \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") "
Oct 30 22:03:31.504393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.504258    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\") pod \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\" (UID: \"213f219c-be97-4ec7-8edb-db3a9a4f3bc0\") "
Oct 30 22:03:31.504818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.504791    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca" pod "213f219c-be97-4ec7-8edb-db3a9a4f3bc0" (UID: "213f219c-be97-4ec7-8edb-db3a9a4f3bc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:31.506963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.506928    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl" pod "213f219c-be97-4ec7-8edb-db3a9a4f3bc0" (UID: "213f219c-be97-4ec7-8edb-db3a9a4f3bc0"). InnerVolumeSpecName "kube-api-access-vbzvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:31.507070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.506959    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca" pod "213f219c-be97-4ec7-8edb-db3a9a4f3bc0" (UID: "213f219c-be97-4ec7-8edb-db3a9a4f3bc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:31.607810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.607573    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzvl\" (UniqueName: \"kubernetes.io/projected/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-kube-api-access-vbzvl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:31.607810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.607613    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:31.607810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.607628    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213f219c-be97-4ec7-8edb-db3a9a4f3bc0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:31.689796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.689608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:31.733166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.733120    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:31.733356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.733196    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:31.917715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.917681    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:03:31.918354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.918339    4410 cpu_manager.go:414] "containerMap: removing container" podUID="213f219c-be97-4ec7-8edb-db3a9a4f3bc0" containerName="revoke-keys"
Oct 30 22:03:31.918398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.918357    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213f219c-be97-4ec7-8edb-db3a9a4f3bc0" containerName="revoke-keys"
Oct 30 22:03:31.918480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.918468    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="213f219c-be97-4ec7-8edb-db3a9a4f3bc0" containerName="revoke-keys"
Oct 30 22:03:31.922330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:31.922313    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.007899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.006792    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:03:32.024870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024815    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.024870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024854    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024904    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024935    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm8m\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024956    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.024985    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.025010    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.025041    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.025099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.025057    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126599    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126686    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126736    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm8m\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126801    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126857    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126904    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126958    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.126998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.126985    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.127334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.127311    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.127589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.127565    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.127740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.127714    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.127840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.127737    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.127912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.127856    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.129412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.129389    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.129882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.129850    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.185565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.185383    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.186246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.186216    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm8m\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.196177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.196151    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"
Oct 30 22:03:32.196177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.196166    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w" event={"ID":"213f219c-be97-4ec7-8edb-db3a9a4f3bc0","Type":"ContainerDied","Data":"0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64"}
Oct 30 22:03:32.196361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.196213    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0867933fe0241ac413b78632a614210a35ae748f4166a0229876fd7d320d2c64"
Oct 30 22:03:32.236285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.236264    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:32.733628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.733588    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:32.734087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.733649    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:32.737592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.737567    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:03:32.738825 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:32.738801    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20125850_cbe5_4b46_b1b2_a41236d1eb55.slice/crio-1a6dda6ca4a2cf050f7855c5474c08679d672006dced17b4948d2ab566c2fd6e WatchSource:0}: Error finding container 1a6dda6ca4a2cf050f7855c5474c08679d672006dced17b4948d2ab566c2fd6e: Status 404 returned error can't find the container with id 1a6dda6ca4a2cf050f7855c5474c08679d672006dced17b4948d2ab566c2fd6e
Oct 30 22:03:32.741249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.741214    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:32.741355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.741290    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:32.741355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:32.741329    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:33.204481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.204443    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" event={"ID":"20125850-cbe5-4b46-b1b2-a41236d1eb55","Type":"ContainerStarted","Data":"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"}
Oct 30 22:03:33.204676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.204506    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" event={"ID":"20125850-cbe5-4b46-b1b2-a41236d1eb55","Type":"ContainerStarted","Data":"1a6dda6ca4a2cf050f7855c5474c08679d672006dced17b4948d2ab566c2fd6e"}
Oct 30 22:03:33.239013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.238966    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:33.242605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.240575    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:33.242605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.240642    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:33.265799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.264145    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"]
Oct 30 22:03:33.277213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.276089    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.327299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.327269    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"]
Oct 30 22:03:33.338818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.334605    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:33.338818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.334720    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:33.338818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.336817    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podStartSLOduration=2.336767592 podStartE2EDuration="2.336767592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:33.32473489 +0000 UTC m=+1266.413472693" watchObservedRunningTime="2026-10-30 22:03:33.336767592 +0000 UTC m=+1266.425505396"
Oct 30 22:03:33.342568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.342001    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:33.343173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.342763    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:33.344142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.344114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.344687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.344666    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.354944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.345270    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8cz\" (UniqueName: \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.467570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.467459    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.470998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.470962    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.471223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.471204    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8cz\" (UniqueName: \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.472644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.472497    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.477201 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.477172    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.528672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.528634    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8cz\" (UniqueName: \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\") pod \"maas-api-revoke-sub-bbbc9857-wxb79\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.591402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.591369    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:33.733226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.733147    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:03:33.733226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.733200    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:03:33.990020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:33.989977    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"]
Oct 30 22:03:33.990636 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:33.990602    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f6ce33_cd76_420c_b4ae_75501f0c46a0.slice/crio-ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797 WatchSource:0}: Error finding container ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797: Status 404 returned error can't find the container with id ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797
Oct 30 22:03:34.213180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.213141    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79" event={"ID":"f0f6ce33-cd76-420c-b4ae-75501f0c46a0","Type":"ContainerStarted","Data":"ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797"}
Oct 30 22:03:34.237382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.237305    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:34.237529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.237369    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:34.736968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.736936    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:34.737270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.737244    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:34.738025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:34.738007    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:03:35.221343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:35.221306    4410 generic.go:356] "Generic (PLEG): container finished" podID="f0f6ce33-cd76-420c-b4ae-75501f0c46a0" containerID="6b51ffbe718d2e50014a3f4466a897ec12756d4375272c945c7c160dde86593e" exitCode=0
Oct 30 22:03:35.221844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:35.221352    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79" event={"ID":"f0f6ce33-cd76-420c-b4ae-75501f0c46a0","Type":"ContainerDied","Data":"6b51ffbe718d2e50014a3f4466a897ec12756d4375272c945c7c160dde86593e"}
Oct 30 22:03:35.237179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:35.237155    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:35.237274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:35.237202    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:36.237386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.237354    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:36.237865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.237407    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:36.413904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.413876    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:36.470744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.470692    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:36.513345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.513268    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\") pod \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") "
Oct 30 22:03:36.513345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.513304    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\") pod \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") "
Oct 30 22:03:36.513565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.513440    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\" (UniqueName: \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\") pod \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\" (UID: \"f0f6ce33-cd76-420c-b4ae-75501f0c46a0\") "
Oct 30 22:03:36.513691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.513655    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca" pod "f0f6ce33-cd76-420c-b4ae-75501f0c46a0" (UID: "f0f6ce33-cd76-420c-b4ae-75501f0c46a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:36.515376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.515348    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca" pod "f0f6ce33-cd76-420c-b4ae-75501f0c46a0" (UID: "f0f6ce33-cd76-420c-b4ae-75501f0c46a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:36.515519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.515472    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz" pod "f0f6ce33-cd76-420c-b4ae-75501f0c46a0" (UID: "f0f6ce33-cd76-420c-b4ae-75501f0c46a0"). InnerVolumeSpecName "kube-api-access-lk8cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:36.614924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.614888    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8cz\" (UniqueName: \"kubernetes.io/projected/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-kube-api-access-lk8cz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:36.614924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.614916    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:36.614924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:36.614926    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6ce33-cd76-420c-b4ae-75501f0c46a0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:37.234821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.234788    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79" event={"ID":"f0f6ce33-cd76-420c-b4ae-75501f0c46a0","Type":"ContainerDied","Data":"ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797"}
Oct 30 22:03:37.234969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.234838    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed353fb2522521b2e10a9212a8c79a570f545fbce1a1f5fd9e80149b2f348797"
Oct 30 22:03:37.234969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.234798    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"
Oct 30 22:03:37.236676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.236647    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:37.236921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.236887    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:37.287935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.287856    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"]
Oct 30 22:03:37.288514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.288492    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f0f6ce33-cd76-420c-b4ae-75501f0c46a0" containerName="revoke-keys"
Oct 30 22:03:37.288596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.288517    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f6ce33-cd76-420c-b4ae-75501f0c46a0" containerName="revoke-keys"
Oct 30 22:03:37.288666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.288653    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f6ce33-cd76-420c-b4ae-75501f0c46a0" containerName="revoke-keys"
Oct 30 22:03:37.291891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.291871    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.354798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.349071    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:37.355059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.355040    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:37.371268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.355272    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:37.371486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.358990    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:37.379381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.379350    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"]
Oct 30 22:03:37.401915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.401890    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"]
Oct 30 22:03:37.405865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.405847    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.422710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.422685    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pd54\" (UniqueName: \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.422833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.422728    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.422935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.422913    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.477113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.477076    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"]
Oct 30 22:03:37.524304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524266    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pd54\" (UniqueName: \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.524466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524334    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.524466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524395    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfss\" (UniqueName: \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.524595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524498    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.524595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524543    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.524702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.524610    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.525086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.525065    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.526923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.526903    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.581199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.581134    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pd54\" (UniqueName: \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\") pod \"maas-api-revoke-sub-b83534b8-6f8tl\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.604611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.604579    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:37.625554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.625523    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.625663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.625586    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.625734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.625706    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfss\" (UniqueName: \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.626337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.626314    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.627851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.627765    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.683767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.683463    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfss\" (UniqueName: \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\") pod \"maas-api-revoke-sub-370051c0-sgpn9\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.720249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.720219    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:37.999479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:37.999454    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"]
Oct 30 22:03:38.001071 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:38.001039    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c0374b_c9ef_4064_857f_130b03a5517e.slice/crio-6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372 WatchSource:0}: Error finding container 6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372: Status 404 returned error can't find the container with id 6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372
Oct 30 22:03:38.113123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:38.113099    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"]
Oct 30 22:03:38.170657 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:38.170415    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880389eb_3712_40d6_9d86_7aa4fed801aa.slice/crio-b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac WatchSource:0}: Error finding container b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac: Status 404 returned error can't find the container with id b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac
Oct 30 22:03:38.238038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:38.238009    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:38.238133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:38.238065    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:38.242294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:38.242265    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9" event={"ID":"880389eb-3712-40d6-9d86-7aa4fed801aa","Type":"ContainerStarted","Data":"b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac"}
Oct 30 22:03:38.243485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:38.243462    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl" event={"ID":"01c0374b-c9ef-4064-857f-130b03a5517e","Type":"ContainerStarted","Data":"6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372"}
Oct 30 22:03:39.236925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.236891    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:03:39.237248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.236945    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:03:39.251033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.251000    4410 generic.go:356] "Generic (PLEG): container finished" podID="880389eb-3712-40d6-9d86-7aa4fed801aa" containerID="8e93cfe997b8eceb0b90454e001370c8ac16814b30a197b191de4116d6a74d27" exitCode=0
Oct 30 22:03:39.251135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.251037    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9" event={"ID":"880389eb-3712-40d6-9d86-7aa4fed801aa","Type":"ContainerDied","Data":"8e93cfe997b8eceb0b90454e001370c8ac16814b30a197b191de4116d6a74d27"}
Oct 30 22:03:39.252855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.252823    4410 generic.go:356] "Generic (PLEG): container finished" podID="01c0374b-c9ef-4064-857f-130b03a5517e" containerID="e8018a9bab3dffae9dceb5ada16a76f520138efb3ec067ea5ec30f9ed36be976" exitCode=0
Oct 30 22:03:39.252949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:39.252891    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl" event={"ID":"01c0374b-c9ef-4064-857f-130b03a5517e","Type":"ContainerDied","Data":"e8018a9bab3dffae9dceb5ada16a76f520138efb3ec067ea5ec30f9ed36be976"}
Oct 30 22:03:40.240867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.240836    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:40.241367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.241200    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:40.242010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.241992    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:03:40.464268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.464235    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:40.497840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.497755    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:40.559446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559410    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\") pod \"01c0374b-c9ef-4064-857f-130b03a5517e\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") "
Oct 30 22:03:40.559682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559485    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\") pod \"01c0374b-c9ef-4064-857f-130b03a5517e\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") "
Oct 30 22:03:40.559682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559599    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\") pod \"880389eb-3712-40d6-9d86-7aa4fed801aa\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") "
Oct 30 22:03:40.559915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559717    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\" (UniqueName: \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\") pod \"880389eb-3712-40d6-9d86-7aa4fed801aa\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") "
Oct 30 22:03:40.559915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559750    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\") pod \"880389eb-3712-40d6-9d86-7aa4fed801aa\" (UID: \"880389eb-3712-40d6-9d86-7aa4fed801aa\") "
Oct 30 22:03:40.559915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559849    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\" (UniqueName: \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\") pod \"01c0374b-c9ef-4064-857f-130b03a5517e\" (UID: \"01c0374b-c9ef-4064-857f-130b03a5517e\") "
Oct 30 22:03:40.559915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.559850    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca" pod "01c0374b-c9ef-4064-857f-130b03a5517e" (UID: "01c0374b-c9ef-4064-857f-130b03a5517e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.561013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.560758    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca" pod "880389eb-3712-40d6-9d86-7aa4fed801aa" (UID: "880389eb-3712-40d6-9d86-7aa4fed801aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.561544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.561311    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.563480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.563452    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca" pod "01c0374b-c9ef-4064-857f-130b03a5517e" (UID: "01c0374b-c9ef-4064-857f-130b03a5517e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:40.564844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.564809    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54" pod "01c0374b-c9ef-4064-857f-130b03a5517e" (UID: "01c0374b-c9ef-4064-857f-130b03a5517e"). InnerVolumeSpecName "kube-api-access-7pd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.564927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.564860    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss" pod "880389eb-3712-40d6-9d86-7aa4fed801aa" (UID: "880389eb-3712-40d6-9d86-7aa4fed801aa"). InnerVolumeSpecName "kube-api-access-xnfss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.564969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.564929    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca" pod "880389eb-3712-40d6-9d86-7aa4fed801aa" (UID: "880389eb-3712-40d6-9d86-7aa4fed801aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:40.629078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.629047    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:40.662601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.662571    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfss\" (UniqueName: \"kubernetes.io/projected/880389eb-3712-40d6-9d86-7aa4fed801aa-kube-api-access-xnfss\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.662601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.662601    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.662601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.662611    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pd54\" (UniqueName: \"kubernetes.io/projected/01c0374b-c9ef-4064-857f-130b03a5517e-kube-api-access-7pd54\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.662830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.662621    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c0374b-c9ef-4064-857f-130b03a5517e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.662830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.662630    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880389eb-3712-40d6-9d86-7aa4fed801aa-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.827073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:40.826965    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:41.265182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.265152    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"
Oct 30 22:03:41.265590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.265182    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9" event={"ID":"880389eb-3712-40d6-9d86-7aa4fed801aa","Type":"ContainerDied","Data":"b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac"}
Oct 30 22:03:41.265590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.265260    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29a552324438e9ce586bf703e85176397da76c0899ee2f8bac230ece6f4a9ac"
Oct 30 22:03:41.266636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.266616    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"
Oct 30 22:03:41.266732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.266661    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl" event={"ID":"01c0374b-c9ef-4064-857f-130b03a5517e","Type":"ContainerDied","Data":"6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372"}
Oct 30 22:03:41.266732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.266710    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6396a67714f880d75ec1293c271c253d8a480449dfcd299d693fc7db3c2c5372"
Oct 30 22:03:41.622277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:41.622192    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:46.571797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:46.571342    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:47.079900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.079856    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"]
Oct 30 22:03:47.117798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.113041    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc19571-jzxnb"]
Oct 30 22:03:47.495799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.493039    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:03:47.612801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.612175    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.644796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.643880    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.703797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.702889    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.872850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.871867    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.885539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:47.885499    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.158916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.158867    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.211799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.211054    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"]
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225360    4410 cpu_manager.go:414] "containerMap: removing container" podUID="01c0374b-c9ef-4064-857f-130b03a5517e" containerName="revoke-keys"
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225392    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c0374b-c9ef-4064-857f-130b03a5517e" containerName="revoke-keys"
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225408    4410 cpu_manager.go:414] "containerMap: removing container" podUID="880389eb-3712-40d6-9d86-7aa4fed801aa" containerName="revoke-keys"
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225415    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880389eb-3712-40d6-9d86-7aa4fed801aa" containerName="revoke-keys"
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225604    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c0374b-c9ef-4064-857f-130b03a5517e" containerName="revoke-keys"
Oct 30 22:03:48.227798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.225618    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="880389eb-3712-40d6-9d86-7aa4fed801aa" containerName="revoke-keys"
Oct 30 22:03:48.242798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.229508    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.279564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.279230    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.282848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.282819    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"]
Oct 30 22:03:48.293812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.293134    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:48.293812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.293210    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:48.323072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.322664    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.323072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.322759    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64l2f\" (UniqueName: \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.323072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.322811    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.349266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.349229    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:48.349700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.349680    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:03:48.373716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.373581    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.424509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.424401    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.424858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.424815    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l2f\" (UniqueName: \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.425053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.425032    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.452532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.450706    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.452532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.451248    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.511824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.502034    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.511824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.505884    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.511824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.506407    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:03:48.550914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.550847    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l2f\" (UniqueName: \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\") pod \"maas-api-revoke-sub-61a92e35-j8zwz\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.596056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.595797    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:48.631617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.631579    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.640390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.640367    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:03:48.669240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.669202    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.693089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.693060    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:03:48.759818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.708844    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.759818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.709212    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:03:48.760615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.760585    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:03:48.895803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.886210    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.895803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:48.886719    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:03:49.299803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.289940    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:03:49.439808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.438603    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d933ccfc-20f8-4805-af85-6edd50130923" path="/var/lib/kubelet/pods/d933ccfc-20f8-4805-af85-6edd50130923/volumes"
Oct 30 22:03:49.492826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.466942    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.507011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.506966    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.549928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.549833    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:49.576799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.572377    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:03:49.576799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.572730    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:03:49.822909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.822817    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:49.823482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.823444    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.831816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.825072    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 22:03:49.858449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.855038    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.871599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.871113    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.894067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:49.894021    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.197796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.197040    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.303799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.293714    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:03:50.412696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.412666    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:03:50.426058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.418790    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.523845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.522132    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:03:50.620841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.614377    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:03:50.620841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.618896    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"]
Oct 30 22:03:50.683003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.682162    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:03:50.701326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.701251    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:03:50.746256 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:03:50.746179    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:03:50.858716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.857980    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.870793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.869409    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:03:50.879193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.879172    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:03:50.893766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.893729    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz" event={"ID":"bd871345-7092-48ab-bded-236815de68ea","Type":"ContainerStarted","Data":"cbf978fe0c53a62c9c10ad4db061b64384be5118d524956b1b3cd991f2b2e3c6"}
Oct 30 22:03:50.910825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.908837    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:50.910825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:50.908888    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:51.052583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.052533    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"]
Oct 30 22:03:51.059617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.059583    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"]
Oct 30 22:03:51.064311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.064286    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc2636e-k7wfd"]
Oct 30 22:03:51.068873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.068850    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-163b6ea4-rnqv5"]
Oct 30 22:03:51.904919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.904888    4410 generic.go:356] "Generic (PLEG): container finished" podID="bd871345-7092-48ab-bded-236815de68ea" containerID="fb59e219d13d042f2676425c9ffbf7fe54b6e2f6b769e7c3162b230e817baa2c" exitCode=0
Oct 30 22:03:51.905309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:51.904930    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz" event={"ID":"bd871345-7092-48ab-bded-236815de68ea","Type":"ContainerDied","Data":"fb59e219d13d042f2676425c9ffbf7fe54b6e2f6b769e7c3162b230e817baa2c"}
Oct 30 22:03:52.426327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.426293    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:03:52.429985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.429968    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.448870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.448842    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.448968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.448882    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.448968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.448904    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449053    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449119    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449174    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjnn\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449202    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449318    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.449561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.449372    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.490214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.490179    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:03:52.550292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550259    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550296    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550322    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550358    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550391    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjnn\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550415    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550491    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550551    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550655    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550685    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.550837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.550798    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.551061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.551039    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.551118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.551096    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.551181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.551169    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.553166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.553138    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.553166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.553157    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.606599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.606559    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.606730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.606618    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjnn\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:52.657245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.657209    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:52.742154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:52.742070    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:53.020451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.020384    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b0ca99-e0ab-45ee-b94c-2dff66dd106e" path="/var/lib/kubelet/pods/34b0ca99-e0ab-45ee-b94c-2dff66dd106e/volumes"
Oct 30 22:03:53.020874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.020858    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680c635c-4ce2-414b-a78e-d05903df01d7" path="/var/lib/kubelet/pods/680c635c-4ce2-414b-a78e-d05903df01d7/volumes"
Oct 30 22:03:53.052615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.052594    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:53.159175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.159150    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\") pod \"bd871345-7092-48ab-bded-236815de68ea\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") "
Oct 30 22:03:53.159322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.159207    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\" (UniqueName: \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\") pod \"bd871345-7092-48ab-bded-236815de68ea\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") "
Oct 30 22:03:53.159322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.159234    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\") pod \"bd871345-7092-48ab-bded-236815de68ea\" (UID: \"bd871345-7092-48ab-bded-236815de68ea\") "
Oct 30 22:03:53.159567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.159544    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca" pod "bd871345-7092-48ab-bded-236815de68ea" (UID: "bd871345-7092-48ab-bded-236815de68ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:53.161273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.161248    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca" pod "bd871345-7092-48ab-bded-236815de68ea" (UID: "bd871345-7092-48ab-bded-236815de68ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:53.161368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.161305    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f" pod "bd871345-7092-48ab-bded-236815de68ea" (UID: "bd871345-7092-48ab-bded-236815de68ea"). InnerVolumeSpecName "kube-api-access-64l2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:53.175063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.175037    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:03:53.176724 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:03:53.176698    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708c45d4_5476_4f69_8449_121ffc27c1f7.slice/crio-65d6dfa66da5cbc4b9d69720cfd7a37166edcbbadeaa12978b71568c46fbc398 WatchSource:0}: Error finding container 65d6dfa66da5cbc4b9d69720cfd7a37166edcbbadeaa12978b71568c46fbc398: Status 404 returned error can't find the container with id 65d6dfa66da5cbc4b9d69720cfd7a37166edcbbadeaa12978b71568c46fbc398
Oct 30 22:03:53.179591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.179556    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:53.179655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.179636    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:53.179698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.179677    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:53.260870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.260839    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd871345-7092-48ab-bded-236815de68ea-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.260870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.260869    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64l2f\" (UniqueName: \"kubernetes.io/projected/bd871345-7092-48ab-bded-236815de68ea-kube-api-access-64l2f\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.261052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.260881    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd871345-7092-48ab-bded-236815de68ea-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.918405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.918366    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" event={"ID":"708c45d4-5476-4f69-8449-121ffc27c1f7","Type":"ContainerStarted","Data":"0c24db8e45d5fe164eb159be46497cd3f0372c7cb5a35e35e60d000c79da8bd9"}
Oct 30 22:03:53.918604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.918427    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" event={"ID":"708c45d4-5476-4f69-8449-121ffc27c1f7","Type":"ContainerStarted","Data":"65d6dfa66da5cbc4b9d69720cfd7a37166edcbbadeaa12978b71568c46fbc398"}
Oct 30 22:03:53.919686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.919664    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"
Oct 30 22:03:53.919807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.919665    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz" event={"ID":"bd871345-7092-48ab-bded-236815de68ea","Type":"ContainerDied","Data":"cbf978fe0c53a62c9c10ad4db061b64384be5118d524956b1b3cd991f2b2e3c6"}
Oct 30 22:03:53.919860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:53.919811    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf978fe0c53a62c9c10ad4db061b64384be5118d524956b1b3cd991f2b2e3c6"
Oct 30 22:03:54.039750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:54.039692    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podStartSLOduration=2.039678357 podStartE2EDuration="2.039678357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:54.038070547 +0000 UTC m=+1287.126808354" watchObservedRunningTime="2026-10-30 22:03:54.039678357 +0000 UTC m=+1287.128416219"
Oct 30 22:03:54.742623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:54.742594    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:03:54.743811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:54.743790    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:54.743901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:54.743835    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:03:55.742626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:55.742587    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:55.743046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:55.742648    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:03:56.742638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:56.742606    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.743034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:56.742661    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:03:57.742869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:57.742836    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.743231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:57.742894    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:03:57.971984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:57.967649    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:58.077162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:58.076861    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"]
Oct 30 22:03:58.079938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:58.079913    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be618162-l9f2v"]
Oct 30 22:03:58.743354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:58.743321    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.743730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:58.743373    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:03:59.019141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:59.019063    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa7cc05-0020-4d59-86ec-ae9e4590c01a" path="/var/lib/kubelet/pods/afa7cc05-0020-4d59-86ec-ae9e4590c01a/volumes"
Oct 30 22:03:59.742968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:59.742934    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.743114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:03:59.742987    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 22:04:00.700115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.700083    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"]
Oct 30 22:04:00.700606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.700591    4410 cpu_manager.go:414] "containerMap: removing container" podUID="bd871345-7092-48ab-bded-236815de68ea" containerName="revoke-keys"
Oct 30 22:04:00.700650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.700607    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd871345-7092-48ab-bded-236815de68ea" containerName="revoke-keys"
Oct 30 22:04:00.700710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.700701    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd871345-7092-48ab-bded-236815de68ea" containerName="revoke-keys"
Oct 30 22:04:00.705287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.705271    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.747099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.747074    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:04:00.747354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.747336    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:04:00.747882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.747867    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:04:00.752869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.752847    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.752960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.752892    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlp9\" (UniqueName: \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.753033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.753017    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.761705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.761659    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"]
Oct 30 22:04:00.807071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.807042    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:00.807183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.807143    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:00.807246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.807213    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:00.807246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.807240    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:04:00.853991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.853961    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.854124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.854032    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlp9\" (UniqueName: \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.854124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.854110    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.854802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.854757    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.856415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.856397    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:00.910179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:00.910154    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlp9\" (UniqueName: \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\") pod \"maas-api-revoke-sub-cdba4ee8-grksg\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:01.017860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:01.017786    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:01.407699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:01.407672    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"]
Oct 30 22:04:01.409223 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:01.409193    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7d5fa2_0182_4213_9d3e_a34ba1f907db.slice/crio-5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8 WatchSource:0}: Error finding container 5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8: Status 404 returned error can't find the container with id 5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8
Oct 30 22:04:02.001747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:02.001663    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg" event={"ID":"ac7d5fa2-0182-4213-9d3e-a34ba1f907db","Type":"ContainerStarted","Data":"6ca99f07a3eeead1910a1c50ee2b9f24aea14644ae3bc36d7ea234f9e072750a"}
Oct 30 22:04:02.001747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:02.001723    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg" event={"ID":"ac7d5fa2-0182-4213-9d3e-a34ba1f907db","Type":"ContainerStarted","Data":"5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8"}
Oct 30 22:04:02.124648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:02.124602    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg" podStartSLOduration=2.12458927 podStartE2EDuration="2.12458927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:02.121584055 +0000 UTC m=+1295.210321857" watchObservedRunningTime="2026-10-30 22:04:02.12458927 +0000 UTC m=+1295.213327072"
Oct 30 22:04:03.008965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:03.008931    4410 generic.go:356] "Generic (PLEG): container finished" podID="ac7d5fa2-0182-4213-9d3e-a34ba1f907db" containerID="6ca99f07a3eeead1910a1c50ee2b9f24aea14644ae3bc36d7ea234f9e072750a" exitCode=0
Oct 30 22:04:03.009381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:03.009012    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg" event={"ID":"ac7d5fa2-0182-4213-9d3e-a34ba1f907db","Type":"ContainerDied","Data":"6ca99f07a3eeead1910a1c50ee2b9f24aea14644ae3bc36d7ea234f9e072750a"}
Oct 30 22:04:03.047612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:03.047569    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:03.225080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:03.225039    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:04.166584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.166552    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:04.294581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.294507    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\" (UniqueName: \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\") pod \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") "
Oct 30 22:04:04.294581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.294576    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\") pod \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") "
Oct 30 22:04:04.294811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.294681    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\") pod \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\" (UID: \"ac7d5fa2-0182-4213-9d3e-a34ba1f907db\") "
Oct 30 22:04:04.295142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.295087    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca" pod "ac7d5fa2-0182-4213-9d3e-a34ba1f907db" (UID: "ac7d5fa2-0182-4213-9d3e-a34ba1f907db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:04.295293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.295269    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.296895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.296867    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca" pod "ac7d5fa2-0182-4213-9d3e-a34ba1f907db" (UID: "ac7d5fa2-0182-4213-9d3e-a34ba1f907db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:04.297185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.297166    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9" pod "ac7d5fa2-0182-4213-9d3e-a34ba1f907db" (UID: "ac7d5fa2-0182-4213-9d3e-a34ba1f907db"). InnerVolumeSpecName "kube-api-access-2tlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:04.396411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.396383    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlp9\" (UniqueName: \"kubernetes.io/projected/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-kube-api-access-2tlp9\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.396411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:04.396410    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d5fa2-0182-4213-9d3e-a34ba1f907db-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.022650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.022621    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"
Oct 30 22:04:05.022822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.022663    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg" event={"ID":"ac7d5fa2-0182-4213-9d3e-a34ba1f907db","Type":"ContainerDied","Data":"5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8"}
Oct 30 22:04:05.022822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.022711    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4d9e4ab8f7fafa365bf7be0b427db871f0f9ff566d7cdab6ebb9dc8cae19b8"
Oct 30 22:04:05.077634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.077601    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"]
Oct 30 22:04:05.078106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.078091    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ac7d5fa2-0182-4213-9d3e-a34ba1f907db" containerName="revoke-keys"
Oct 30 22:04:05.078155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.078107    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7d5fa2-0182-4213-9d3e-a34ba1f907db" containerName="revoke-keys"
Oct 30 22:04:05.078216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.078206    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7d5fa2-0182-4213-9d3e-a34ba1f907db" containerName="revoke-keys"
Oct 30 22:04:05.081477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.081458    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.136116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.136090    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:05.136230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.136139    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:05.136230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.136182    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:05.136307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.136237    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:04:05.144751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.144728    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"]
Oct 30 22:04:05.205312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.205286    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28gn\" (UniqueName: \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.205683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.205332    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.205683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.205408    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.306877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.306787    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28gn\" (UniqueName: \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.306877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.306848    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.307048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.306891    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.307717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.307688    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.309449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.309427    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.363343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.363322    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28gn\" (UniqueName: \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\") pod \"maas-api-revoke-sub-c511335c-wvncq\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.397537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.397512    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:05.772637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:05.772614    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"]
Oct 30 22:04:05.775064 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:05.775035    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10b7d7b_d47b_4f07_8441_d9fed355ac77.slice/crio-97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35 WatchSource:0}: Error finding container 97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35: Status 404 returned error can't find the container with id 97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35
Oct 30 22:04:06.030755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:06.030675    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq" event={"ID":"d10b7d7b-d47b-4f07-8441-d9fed355ac77","Type":"ContainerStarted","Data":"97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35"}
Oct 30 22:04:07.037550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.037518    4410 generic.go:356] "Generic (PLEG): container finished" podID="d10b7d7b-d47b-4f07-8441-d9fed355ac77" containerID="5d977a55e963786710fa61529d4fa0727e228ff184eb7c37b16087267513c902" exitCode=0
Oct 30 22:04:07.037550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.037550    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq" event={"ID":"d10b7d7b-d47b-4f07-8441-d9fed355ac77","Type":"ContainerDied","Data":"5d977a55e963786710fa61529d4fa0727e228ff184eb7c37b16087267513c902"}
Oct 30 22:04:07.263447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.263409    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:04:07.267522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.267505    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329295    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhphj\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329409    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329494    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329541    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329561    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329645    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329685    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329755    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.329823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.329805    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.367710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.367673    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:04:07.430597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430556    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.430808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430642    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.430808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430662    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.430808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430683    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.430808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430738    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.430808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430790    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430857    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430893    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.430950    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhphj\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.431056    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.431069    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.431144    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.431258    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.431877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.431851    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.433388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.433365    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.433766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.433743    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.488160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.488129    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.488343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.488322    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhphj\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:07.587983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:07.587920    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:08.012439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.012413    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:04:08.013594 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:08.013568    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a4c55f_2574_49aa_a3c5_17d775c86e8c.slice/crio-033fc558c5a915d82c957fd5325b39195a82f8ae38d0f215cd2f34dd658bbff6 WatchSource:0}: Error finding container 033fc558c5a915d82c957fd5325b39195a82f8ae38d0f215cd2f34dd658bbff6: Status 404 returned error can't find the container with id 033fc558c5a915d82c957fd5325b39195a82f8ae38d0f215cd2f34dd658bbff6
Oct 30 22:04:08.016048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.016019    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:08.016118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.016078    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:08.016118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.016110    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:08.045623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.045592    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" event={"ID":"c6a4c55f-2574-49aa-a3c5-17d775c86e8c","Type":"ContainerStarted","Data":"033fc558c5a915d82c957fd5325b39195a82f8ae38d0f215cd2f34dd658bbff6"}
Oct 30 22:04:08.089918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.089890    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:08.200369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.200341    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:08.348723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.348640    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\") pod \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") "
Oct 30 22:04:08.348889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.348873    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\") pod \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") "
Oct 30 22:04:08.349001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.348986    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\" (UniqueName: \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\") pod \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\" (UID: \"d10b7d7b-d47b-4f07-8441-d9fed355ac77\") "
Oct 30 22:04:08.349361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.349294    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca" pod "d10b7d7b-d47b-4f07-8441-d9fed355ac77" (UID: "d10b7d7b-d47b-4f07-8441-d9fed355ac77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:08.349583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.349563    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:08.351382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.351359    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca" pod "d10b7d7b-d47b-4f07-8441-d9fed355ac77" (UID: "d10b7d7b-d47b-4f07-8441-d9fed355ac77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:08.351497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.351454    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn" pod "d10b7d7b-d47b-4f07-8441-d9fed355ac77" (UID: "d10b7d7b-d47b-4f07-8441-d9fed355ac77"). InnerVolumeSpecName "kube-api-access-t28gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:08.450842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.450806    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28gn\" (UniqueName: \"kubernetes.io/projected/d10b7d7b-d47b-4f07-8441-d9fed355ac77-kube-api-access-t28gn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:08.450842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:08.450835    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10b7d7b-d47b-4f07-8441-d9fed355ac77-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.076723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.076692    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"
Oct 30 22:04:09.077206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.076733    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq" event={"ID":"d10b7d7b-d47b-4f07-8441-d9fed355ac77","Type":"ContainerDied","Data":"97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35"}
Oct 30 22:04:09.077206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.076824    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97f90da717a38527409688667026188c902a7b3988498d2d44f87fd7cea47c35"
Oct 30 22:04:09.078607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.078570    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" event={"ID":"c6a4c55f-2574-49aa-a3c5-17d775c86e8c","Type":"ContainerStarted","Data":"700b66ff98b3f054322bc1289632f5f9d30d08507c19927806ec27e03885243d"}
Oct 30 22:04:09.295383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.295269    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podStartSLOduration=2.295249688 podStartE2EDuration="2.295249688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:09.244861425 +0000 UTC m=+1302.333599239" watchObservedRunningTime="2026-10-30 22:04:09.295249688 +0000 UTC m=+1302.383987493"
Oct 30 22:04:09.588137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.588047    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:09.589625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.589599    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:09.589752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:09.589644    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:10.571396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.571355    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:04:10.572096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.572074    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d10b7d7b-d47b-4f07-8441-d9fed355ac77" containerName="revoke-keys"
Oct 30 22:04:10.572185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.572096    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b7d7b-d47b-4f07-8441-d9fed355ac77" containerName="revoke-keys"
Oct 30 22:04:10.572250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.572237    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b7d7b-d47b-4f07-8441-d9fed355ac77" containerName="revoke-keys"
Oct 30 22:04:10.575493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.575468    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.588629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.588604    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:10.588735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.588655    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:10.688598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.688560    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.688752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.688628    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.688827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.688793    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.688939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.688919    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.689010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.688966    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.689109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.689078    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.689801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.689177    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.689801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.689239    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.689801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.689348    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4kg\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.695193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.695169    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:04:10.793388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.793338    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4kg\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.793634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.793616    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.793809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.793790    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.793974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.793960    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.794135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.794121    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.794236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.794224    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.794399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.794386    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.794517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.794503    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.794632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.794619    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.795521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.795495    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.796574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.796518    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.797684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.796923    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.797684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.797242    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.798399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.798371    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.800304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.800283    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.805014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.804989    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.872316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.872241    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4kg\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.872316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.872274    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:10.888664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:10.888636    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:11.341710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.341681    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:04:11.343214 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:11.343187    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a85a03_e3a6_4c45_bce9_36b6bd4193a2.slice/crio-a51af0da2ba2425f13d2cbef40127b8b13911cf1f8d388d9b5a72d866827ea40 WatchSource:0}: Error finding container a51af0da2ba2425f13d2cbef40127b8b13911cf1f8d388d9b5a72d866827ea40: Status 404 returned error can't find the container with id a51af0da2ba2425f13d2cbef40127b8b13911cf1f8d388d9b5a72d866827ea40
Oct 30 22:04:11.371313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.369980    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:11.371313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.370068    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:11.371313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.370110    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:11.521691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.519061    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"]
Oct 30 22:04:11.524020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.523997    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.577562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.577530    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:11.578356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.578325    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:11.579059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.579036    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:11.579164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.579144    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:04:11.604678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.595631    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"]
Oct 30 22:04:11.605869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.598885    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:11.606071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.606042    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:11.707183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.707150    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2fg\" (UniqueName: \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.707423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.707407    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.707733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.707700    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.809684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.809648    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.809893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.809864    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2fg\" (UniqueName: \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.809977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.809957    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.810935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.810892    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.819142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.819117    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.888843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.887161    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2fg\" (UniqueName: \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\") pod \"maas-api-revoke-sub-d2c5ed83-mp5jc\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:11.888843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:11.887384    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:12.683473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.682946    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.683473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.683031    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:12.712543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.711456    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" event={"ID":"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2","Type":"ContainerStarted","Data":"0a9e5c4e79e95b423d5cefdca61cea8f611d581078204c1c9fd19ca7878b1340"}
Oct 30 22:04:12.712543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.711529    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" event={"ID":"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2","Type":"ContainerStarted","Data":"a51af0da2ba2425f13d2cbef40127b8b13911cf1f8d388d9b5a72d866827ea40"}
Oct 30 22:04:12.861798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.820478    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"]
Oct 30 22:04:12.903726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.903691    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:12.909354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.904787    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.909354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.904846    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:12.959170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:12.958990    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podStartSLOduration=2.923819758 podStartE2EDuration="2.923819758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:12.882070474 +0000 UTC m=+1305.970808303" watchObservedRunningTime="2026-10-30 22:04:12.923819758 +0000 UTC m=+1306.012557562"
Oct 30 22:04:13.286965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.286909    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.300360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.300305    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.375093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.374633    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.375093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.375024    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.424855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.424173    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.533859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.532877    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.595800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.590234    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.595800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.590313    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:13.709799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.709730    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.773796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.771690    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.798852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.791452    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc" event={"ID":"f30c9830-8f97-4b57-ba85-f6a0267369e4","Type":"ContainerStarted","Data":"7bb79bc0eb3d08195e652863dc342a432fbf8b95026278397cef915f9b3ac55c"}
Oct 30 22:04:13.798852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.791562    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc" event={"ID":"f30c9830-8f97-4b57-ba85-f6a0267369e4","Type":"ContainerStarted","Data":"c478ee7b44c22db51b39e2bc13cc5d32f53833535450618355518d2e9595e7fc"}
Oct 30 22:04:13.830798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.829948    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.874798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.874660    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.894799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.893639    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.906797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.904123    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.938815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.924035    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.938815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.924102    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:13.946158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:13.946091    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc" podStartSLOduration=2.946069578 podStartE2EDuration="2.946069578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:13.94349408 +0000 UTC m=+1307.032231886" watchObservedRunningTime="2026-10-30 22:04:13.946069578 +0000 UTC m=+1307.034807382"
Oct 30 22:04:14.190797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.189947    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.311004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.308913    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.311004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.309441    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:04:14.410799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.383862    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.410799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.384278    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 22:04:14.458946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.458850    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.465944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.465915    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:04:14.533999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.533957    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.540801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.534565    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:04:14.651800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.649187    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.651800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.649249    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:14.775534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.775415    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.781133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.781101    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:04:14.839809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.830503    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.905134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.905090    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.906004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.871934    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:04:14.909909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.909886    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:04:14.934798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.905671    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.935429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.935403    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:04:14.977813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.975753    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.977813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:14.975831    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:15.006495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.006443    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:15.006804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.006749    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.073168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.073063    4410 generic.go:356] "Generic (PLEG): container finished" podID="f30c9830-8f97-4b57-ba85-f6a0267369e4" containerID="7bb79bc0eb3d08195e652863dc342a432fbf8b95026278397cef915f9b3ac55c" exitCode=0
Oct 30 22:04:15.158800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.134339    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc" event={"ID":"f30c9830-8f97-4b57-ba85-f6a0267369e4","Type":"ContainerDied","Data":"7bb79bc0eb3d08195e652863dc342a432fbf8b95026278397cef915f9b3ac55c"}
Oct 30 22:04:15.208888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.208848    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.209652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.209630    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:04:15.337850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.309184    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.391912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.391865    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.464910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.464857    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.536698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.535868    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.693853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.674425    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.693853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.674482    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:15.732696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.732577    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:15.785811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.781863    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.980908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.894264    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.980908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.897925    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.980908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.897969    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:15.980908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.909874    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.980908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:15.964012    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.124799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.077651    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.195386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.195347    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.195687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.195641    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.216495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.216112    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.459797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.455879    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.477809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.475897    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.482800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.479591    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.551986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.551886    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.605111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.602894    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.605111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.602967    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:16.798799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.785864    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.806359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.805523    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.806359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.805586    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.825810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.824729    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:04:16.877968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.877936    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:04:16.895877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.895836    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.896433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.896413    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:04:16.897125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.896761    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.903906 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.900018    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.904092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.904067    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:16.971124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.971094    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:04:16.994274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.994242    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:04:16.995207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.995150    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 22:04:16.995403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:16.995388    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:04:17.000310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.000293    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:04:17.002349 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.002223    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:04:17.010942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.010920    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:04:17.032452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.032279    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:04:17.115115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.115090    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:17.207160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.207121    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\") pod \"f30c9830-8f97-4b57-ba85-f6a0267369e4\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") "
Oct 30 22:04:17.207356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.207178    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\") pod \"f30c9830-8f97-4b57-ba85-f6a0267369e4\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") "
Oct 30 22:04:17.207432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.207401    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\" (UniqueName: \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\") pod \"f30c9830-8f97-4b57-ba85-f6a0267369e4\" (UID: \"f30c9830-8f97-4b57-ba85-f6a0267369e4\") "
Oct 30 22:04:17.207499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.207468    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca" pod "f30c9830-8f97-4b57-ba85-f6a0267369e4" (UID: "f30c9830-8f97-4b57-ba85-f6a0267369e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:17.207905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.207874    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:17.209453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.209414    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca" pod "f30c9830-8f97-4b57-ba85-f6a0267369e4" (UID: "f30c9830-8f97-4b57-ba85-f6a0267369e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:17.210150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.210119    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg" pod "f30c9830-8f97-4b57-ba85-f6a0267369e4" (UID: "f30c9830-8f97-4b57-ba85-f6a0267369e4"). InnerVolumeSpecName "kube-api-access-dq2fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:17.282819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.281754    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"
Oct 30 22:04:17.282819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.281968    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc" event={"ID":"f30c9830-8f97-4b57-ba85-f6a0267369e4","Type":"ContainerDied","Data":"c478ee7b44c22db51b39e2bc13cc5d32f53833535450618355518d2e9595e7fc"}
Oct 30 22:04:17.282819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.282050    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c478ee7b44c22db51b39e2bc13cc5d32f53833535450618355518d2e9595e7fc"
Oct 30 22:04:17.309578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.309540    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2fg\" (UniqueName: \"kubernetes.io/projected/f30c9830-8f97-4b57-ba85-f6a0267369e4-kube-api-access-dq2fg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:17.309578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.309576    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30c9830-8f97-4b57-ba85-f6a0267369e4-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:17.588522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.588469    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.588688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.588543    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:04:17.889512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.889480    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.889926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:17.889531    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:18.634372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:18.634337    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:18.889990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:18.889957    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.890387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:18.890024    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:19.629910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:19.629869    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:19.630098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:19.629946    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:19.895183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:19.895150    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.895687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:19.895211    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:20.122692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.122656    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:20.122986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.122961    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:20.123562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.123544    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:04:20.284861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.284754    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:20.534131 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:04:20.534092    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:04:20.657724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.657591    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:20.889584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.889542    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.889810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.889596    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 22:04:20.892387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:20.892348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:21.893788 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:21.893741    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:21.894245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:21.894145    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:21.894839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:21.894815    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:04:21.958141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:21.958108    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:24.799978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.799944    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"]
Oct 30 22:04:24.800414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.800399    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f30c9830-8f97-4b57-ba85-f6a0267369e4" containerName="revoke-keys"
Oct 30 22:04:24.800414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.800414    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30c9830-8f97-4b57-ba85-f6a0267369e4" containerName="revoke-keys"
Oct 30 22:04:24.800520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.800510    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30c9830-8f97-4b57-ba85-f6a0267369e4" containerName="revoke-keys"
Oct 30 22:04:24.803903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.803879    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:24.857193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.857167    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:24.857321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.857294    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:04:24.861516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.861495    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"]
Oct 30 22:04:24.901554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.901519    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:24.901710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.901641    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:24.901785 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.901749    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfkr\" (UniqueName: \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:24.906835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.906805    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:24.906947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:24.906888    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:25.003236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.003204    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.003393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.003306    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.003453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.003411    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfkr\" (UniqueName: \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.004110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.004088    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.005830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.005807    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.059953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.059892    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfkr\" (UniqueName: \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\") pod \"maas-api-revoke-sub-3626c1fd-sp8cw\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.116758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.116738    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:25.491650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:25.491621    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"]
Oct 30 22:04:25.492667 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:25.492637    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435d7350_72ab_4fc3_a7d5_58e1e464cbb9.slice/crio-718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0 WatchSource:0}: Error finding container 718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0: Status 404 returned error can't find the container with id 718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0
Oct 30 22:04:26.355366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:26.355330    4410 generic.go:356] "Generic (PLEG): container finished" podID="435d7350-72ab-4fc3-a7d5-58e1e464cbb9" containerID="7e8b82d2df7a73545f3fca00b0193d954d2fc1e68749ea641fe82a20772ddb54" exitCode=0
Oct 30 22:04:26.355757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:26.355416    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw" event={"ID":"435d7350-72ab-4fc3-a7d5-58e1e464cbb9","Type":"ContainerDied","Data":"7e8b82d2df7a73545f3fca00b0193d954d2fc1e68749ea641fe82a20772ddb54"}
Oct 30 22:04:26.355757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:26.355499    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw" event={"ID":"435d7350-72ab-4fc3-a7d5-58e1e464cbb9","Type":"ContainerStarted","Data":"718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0"}
Oct 30 22:04:26.961766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:26.961722    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:27.242831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.242744    4410 scope.go:122] "RemoveContainer" containerID="3db9b28d30108c65261f99dee0888d24b5fcb9468c97eaef8eaf48d0ec06d593"
Oct 30 22:04:27.251905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.251888    4410 scope.go:122] "RemoveContainer" containerID="4d96b514059ab3a94b53d1e8b9d02326247b60298afa41664efa0978c1a79dde"
Oct 30 22:04:27.262590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.262568    4410 scope.go:122] "RemoveContainer" containerID="3e7d50414ad33675c879994b991f8e0db54e57695dacbffb887af54c36770c84"
Oct 30 22:04:27.270819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.270761    4410 scope.go:122] "RemoveContainer" containerID="175f385514f448fbb605744370294d4b323ac05e285904b47ccb646cc2c3db10"
Oct 30 22:04:27.513556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.513531    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:27.633256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.633222    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\") pod \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") "
Oct 30 22:04:27.633434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.633315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\" (UniqueName: \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\") pod \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") "
Oct 30 22:04:27.633434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.633340    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\") pod \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\" (UID: \"435d7350-72ab-4fc3-a7d5-58e1e464cbb9\") "
Oct 30 22:04:27.633607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.633581    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca" pod "435d7350-72ab-4fc3-a7d5-58e1e464cbb9" (UID: "435d7350-72ab-4fc3-a7d5-58e1e464cbb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:27.633887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.633860    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:27.635438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.635410    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca" pod "435d7350-72ab-4fc3-a7d5-58e1e464cbb9" (UID: "435d7350-72ab-4fc3-a7d5-58e1e464cbb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:27.635594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.635574    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr" pod "435d7350-72ab-4fc3-a7d5-58e1e464cbb9" (UID: "435d7350-72ab-4fc3-a7d5-58e1e464cbb9"). InnerVolumeSpecName "kube-api-access-xgfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:27.735458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.735427    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfkr\" (UniqueName: \"kubernetes.io/projected/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-kube-api-access-xgfkr\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:27.735458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:27.735455    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435d7350-72ab-4fc3-a7d5-58e1e464cbb9-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:28.368914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:28.368881    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"
Oct 30 22:04:28.368914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:28.368891    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw" event={"ID":"435d7350-72ab-4fc3-a7d5-58e1e464cbb9","Type":"ContainerDied","Data":"718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0"}
Oct 30 22:04:28.369120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:28.368948    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718b3d878cec83e802965531e9c7d1addfdc621300d23e5c9b7c0f8d194e94a0"
Oct 30 22:04:31.187115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:31.187084    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:31.878882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:31.878751    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:32.420618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.420585    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:04:32.421092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.421076    4410 cpu_manager.go:414] "containerMap: removing container" podUID="435d7350-72ab-4fc3-a7d5-58e1e464cbb9" containerName="revoke-keys"
Oct 30 22:04:32.421134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.421093    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435d7350-72ab-4fc3-a7d5-58e1e464cbb9" containerName="revoke-keys"
Oct 30 22:04:32.421201 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.421192    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="435d7350-72ab-4fc3-a7d5-58e1e464cbb9" containerName="revoke-keys"
Oct 30 22:04:32.424751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.424736    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490211    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490260    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490376    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490402    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490473    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj5rj\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490507    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490540    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490649    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.490697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.490684    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.530536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.530508    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:04:32.592201 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592162    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592218    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592257    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592294    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592338    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592365    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592423    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj5rj\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592452    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592511    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592614    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592667    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.592833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592712    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.593000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592951    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.593055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.592999    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.594944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.594923    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.595091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.595074    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.650442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.650411    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.650651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.650629    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj5rj\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:32.738071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:32.738023    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:33.167729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.167696    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:04:33.168235 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:33.168208    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703d5a0_297b_4e94_8686_4a490c09cd22.slice/crio-f0918b8301cad02b018018c0b3488129722cc554ff5a446e0b185ebf97885573 WatchSource:0}: Error finding container f0918b8301cad02b018018c0b3488129722cc554ff5a446e0b185ebf97885573: Status 404 returned error can't find the container with id f0918b8301cad02b018018c0b3488129722cc554ff5a446e0b185ebf97885573
Oct 30 22:04:33.170551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.170523    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:33.170604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.170583    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:33.170644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.170629    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:33.402867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.402834    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" event={"ID":"e703d5a0-297b-4e94-8686-4a490c09cd22","Type":"ContainerStarted","Data":"819e48a54ba46336767962644b5937f7f0a6e5ad28e5c24751f44a62db428327"}
Oct 30 22:04:33.403032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.402885    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" event={"ID":"e703d5a0-297b-4e94-8686-4a490c09cd22","Type":"ContainerStarted","Data":"f0918b8301cad02b018018c0b3488129722cc554ff5a446e0b185ebf97885573"}
Oct 30 22:04:33.523769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.523648    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podStartSLOduration=1.523627351 podStartE2EDuration="1.523627351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:33.522274771 +0000 UTC m=+1326.611012577" watchObservedRunningTime="2026-10-30 22:04:33.523627351 +0000 UTC m=+1326.612365155"
Oct 30 22:04:33.739025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:33.738987    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:34.615803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.614994    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:04:34.619697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.619673    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.702283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.702248    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:04:34.714958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.714916    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:34.718681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.718642    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.718970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.718953    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8drm\" (UniqueName: \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.719112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.719097    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.719316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.719296    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.719595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.719580    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.719877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.719858    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.720726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.720701    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"distinct-w2-90e0d5-kserve-self-signed-certs\""
Oct 30 22:04:34.739542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.739516    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.739767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.739743    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:34.828945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.828906    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.829320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.829293    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.829492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.829472    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.829720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.829695    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.829831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.829418    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.830216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.830189    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.830304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.830230    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8drm\" (UniqueName: \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.830304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.830265    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.830571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.830550    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.832688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.832663    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.833936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.833908    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.903256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.903216    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8drm\" (UniqueName: \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\") pod \"distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:34.943427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:34.943210    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:35.334175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.334132    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:35.536603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.536498    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:35.829858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.829696    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.829858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.829747    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:35.850808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.850741    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:04:35.933797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:35.922527    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:36.308469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.308430    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:36.371008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.370858    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:04:36.519698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.519661    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:36.539862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.539765    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:04:36.741795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.741147    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:36.741795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.741214    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:36.794907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.794857    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.867766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.867727    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" event={"ID":"0f301c5e-8f5c-4179-91f3-1f2724216584","Type":"ContainerStarted","Data":"61ef46ae8b18d3c283a0ca675a2c817a8047ccf4d35e6c5f0f803ee21dbe0c63"}
Oct 30 22:04:36.898835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.898791    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.900584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.900552    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.987799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:36.987184    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"]
Oct 30 22:04:37.064830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.058885    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:04:37.064830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.058998    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.064830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.060929    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.064830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.061009    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:04:37.064830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.061702    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.090328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.080862    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.090328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.081140    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"]
Oct 30 22:04:37.131794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.128363    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.173798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.173326    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.173798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.173496    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7px\" (UniqueName: \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.173798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.173615    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.204803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.182120    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:04:37.205016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.191919    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:37.245078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.245045    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:37.245409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.245388    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:37.292804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.274584    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.293368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.293335    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7px\" (UniqueName: \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.297795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.293587    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.298094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.294586    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.324319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.324248    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.374836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.374765    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.375518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.375492    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:04:37.444844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.444191    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7px\" (UniqueName: \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\") pod \"maas-api-revoke-sub-d78f992f-5rsd4\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.551802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.544925    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.551802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.545636    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:04:37.622174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.622093    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:37.791798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.746153    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:37.791798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.746209    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:37.915798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:37.911966    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:04:38.032806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.024614    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:38.033080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.033036    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.033372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.033352    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:04:38.083904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.083860    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" event={"ID":"0f301c5e-8f5c-4179-91f3-1f2724216584","Type":"ContainerStarted","Data":"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"}
Oct 30 22:04:38.114799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.113647    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:04:38.114799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.114079    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.119866    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.120207    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.120495    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.120546    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.131909    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.134800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.132283    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:04:38.335002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.314836    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:04:38.442799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.396977    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.613786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.613727    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.794799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.770154    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:38.799968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.799859    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:38.820934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.795891    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.822426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.812058    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:04:38.822426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.821243    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:04:38.822426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.812165    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:38.838872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.835303    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:04:38.838872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.836595    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:04:38.882126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:38.882064    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" podStartSLOduration=4.882042972 podStartE2EDuration="4.882042972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:38.795389808 +0000 UTC m=+1331.884127612" watchObservedRunningTime="2026-10-30 22:04:38.882042972 +0000 UTC m=+1331.970780776"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.010541    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.011731    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.011800    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.011858    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.011982    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.012056    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcmx\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.012108    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.012232    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.012276    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.012498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.012305    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.022600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.022555    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.090038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.089957    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:39.090038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.090019    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:04:39.090038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.090042    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:04:39.090549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.090070    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:04:39.090549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.090092    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:04:39.113298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113264    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113329    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcmx\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113358    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113416    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113437    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113454    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113492    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113521    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113572    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113720    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.113995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.113972    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.114087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.114038    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.114087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.114062    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.114408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.114383    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.115917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.115884    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.116142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.116121    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.138564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.138530    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"]
Oct 30 22:04:39.139979 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:39.139952    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f04fba_ad84_4390_bec7_4a069953a8aa.slice/crio-abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77 WatchSource:0}: Error finding container abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77: Status 404 returned error can't find the container with id abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77
Oct 30 22:04:39.191844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.187475    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.191844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.189083    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcmx\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.201397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.201373    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:39.610874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.610846    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:04:39.613144 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:04:39.613117    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71c670c_8115_4339_9481_7ac72c34920f.slice/crio-1283f36f21b3f842d98e3199446ced800f6eff2bd9f99c16757b07ecd3428f31 WatchSource:0}: Error finding container 1283f36f21b3f842d98e3199446ced800f6eff2bd9f99c16757b07ecd3428f31: Status 404 returned error can't find the container with id 1283f36f21b3f842d98e3199446ced800f6eff2bd9f99c16757b07ecd3428f31
Oct 30 22:04:39.616332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.616292    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:39.616467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.616376    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:39.616467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.616415    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:39.677292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.677230    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:39.739463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.739345    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:39.739463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:39.739400    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:40.079118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.079037    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" event={"ID":"e71c670c-8115-4339-9481-7ac72c34920f","Type":"ContainerStarted","Data":"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"}
Oct 30 22:04:40.079118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.079102    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" event={"ID":"e71c670c-8115-4339-9481-7ac72c34920f","Type":"ContainerStarted","Data":"1283f36f21b3f842d98e3199446ced800f6eff2bd9f99c16757b07ecd3428f31"}
Oct 30 22:04:40.081980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.081946    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4" event={"ID":"d6f04fba-ad84-4390-bec7-4a069953a8aa","Type":"ContainerStarted","Data":"ebe529d1ff3e3dc60b11a7a3037309e37b02c7d42b86f8e71b8c8fd25807c7dc"}
Oct 30 22:04:40.082115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.081996    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4" event={"ID":"d6f04fba-ad84-4390-bec7-4a069953a8aa","Type":"ContainerStarted","Data":"abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77"}
Oct 30 22:04:40.207689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.205843    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:40.207689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.206862    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:40.207689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.206931    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:40.406053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.405992    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podStartSLOduration=3.405945527 podStartE2EDuration="3.405945527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:40.404125047 +0000 UTC m=+1333.492862852" watchObservedRunningTime="2026-10-30 22:04:40.405945527 +0000 UTC m=+1333.494683331"
Oct 30 22:04:40.532515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.532462    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4" podStartSLOduration=4.532448557 podStartE2EDuration="4.532448557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:40.52891636 +0000 UTC m=+1333.617654174" watchObservedRunningTime="2026-10-30 22:04:40.532448557 +0000 UTC m=+1333.621186408"
Oct 30 22:04:40.584048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.584010    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:40.742929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.739847    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:40.742929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.739903    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:40.798271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:40.797622    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:41.096186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.096093    4410 generic.go:356] "Generic (PLEG): container finished" podID="d6f04fba-ad84-4390-bec7-4a069953a8aa" containerID="ebe529d1ff3e3dc60b11a7a3037309e37b02c7d42b86f8e71b8c8fd25807c7dc" exitCode=0
Oct 30 22:04:41.096760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.096294    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4" event={"ID":"d6f04fba-ad84-4390-bec7-4a069953a8aa","Type":"ContainerDied","Data":"ebe529d1ff3e3dc60b11a7a3037309e37b02c7d42b86f8e71b8c8fd25807c7dc"}
Oct 30 22:04:41.272799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.267844    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:41.272799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.267904    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:41.739199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.739160    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:04:41.739442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:41.739224    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:04:42.206738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.206175    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:42.206738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.206241    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:42.325822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.325797    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:42.502061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.502027    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\") pod \"d6f04fba-ad84-4390-bec7-4a069953a8aa\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") "
Oct 30 22:04:42.502235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.502145    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\" (UniqueName: \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\") pod \"d6f04fba-ad84-4390-bec7-4a069953a8aa\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") "
Oct 30 22:04:42.502323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.502306    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\") pod \"d6f04fba-ad84-4390-bec7-4a069953a8aa\" (UID: \"d6f04fba-ad84-4390-bec7-4a069953a8aa\") "
Oct 30 22:04:42.502485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.502462    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca" pod "d6f04fba-ad84-4390-bec7-4a069953a8aa" (UID: "d6f04fba-ad84-4390-bec7-4a069953a8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:42.502860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.502839    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:42.504399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.504361    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca" pod "d6f04fba-ad84-4390-bec7-4a069953a8aa" (UID: "d6f04fba-ad84-4390-bec7-4a069953a8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:42.504504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.504442    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px" pod "d6f04fba-ad84-4390-bec7-4a069953a8aa" (UID: "d6f04fba-ad84-4390-bec7-4a069953a8aa"). InnerVolumeSpecName "kube-api-access-xj7px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:42.604255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.604175    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj7px\" (UniqueName: \"kubernetes.io/projected/d6f04fba-ad84-4390-bec7-4a069953a8aa-kube-api-access-xj7px\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:42.604255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:42.604216    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f04fba-ad84-4390-bec7-4a069953a8aa-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:43.112417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.112380    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4" event={"ID":"d6f04fba-ad84-4390-bec7-4a069953a8aa","Type":"ContainerDied","Data":"abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77"}
Oct 30 22:04:43.112576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.112433    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf8a8e30fe9ef8e2c36534b1c3a1e88fa1b8adc9e57a7ed710616cfa50dcd77"
Oct 30 22:04:43.112576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.112392    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"
Oct 30 22:04:43.201890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.201858    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:43.202046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.201911    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:43.781796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.780874    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:43.781796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:43.780938    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.070516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.066135    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:44.070516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.067064    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:44.078941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.075989    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:04:44.202164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.202129    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:44.202335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.202188    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:44.944403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.944370    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:44.980892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:44.980862    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:45.136594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:45.136560    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:04:45.202573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:45.202480    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:45.202573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:45.202554    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:45.374712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:45.374674    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:46.202003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:46.201965    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:46.202384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:46.202043    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 22:04:46.766224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:46.766188    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:47.206153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:47.206125    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:47.206614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:47.206597    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:47.207241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:47.207218    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:04:50.522013 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:04:50.521976    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:04:50.647176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:50.647143    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:51.727931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:51.727893    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:55.495745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:55.495708    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:56.196123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:56.196081    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:56.538035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:04:56.537941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:00.449413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:00.449373    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:03.620806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.620756    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:05:03.621277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.621262    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d6f04fba-ad84-4390-bec7-4a069953a8aa" containerName="revoke-keys"
Oct 30 22:05:03.621320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.621277    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f04fba-ad84-4390-bec7-4a069953a8aa" containerName="revoke-keys"
Oct 30 22:05:03.621356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.621346    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f04fba-ad84-4390-bec7-4a069953a8aa" containerName="revoke-keys"
Oct 30 22:05:03.624606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.624586    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.676070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.676040    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\""
Oct 30 22:05:03.682713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682677    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52z9l\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.682829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682736    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.682829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682811    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.682935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682831    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.682935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682860    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.682935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.682878    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.683086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.683048    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.683143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.683113    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.683191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.683151    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.730700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.730667    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:05:03.783934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.783889    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52z9l\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.783949    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.783975    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.783995    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784022    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784045    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784105    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784157    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784189    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784539    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784851    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784881    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.784986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784948    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.785106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.784979    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.786437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.786411    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.786679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.786662    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.841016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.840988    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52z9l\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.841136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.841075    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:03.947243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:03.947212    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:04.376983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:04.376951    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:05:04.377249 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:05:04.377227    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36eb9dc3_6c57_42b7_850e_72c689bb3a05.slice/crio-a555588a58ee2862bdab44a688d01b87c6ec75afc035a66f2e6d117e3fb8bdc9 WatchSource:0}: Error finding container a555588a58ee2862bdab44a688d01b87c6ec75afc035a66f2e6d117e3fb8bdc9: Status 404 returned error can't find the container with id a555588a58ee2862bdab44a688d01b87c6ec75afc035a66f2e6d117e3fb8bdc9
Oct 30 22:05:04.379364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:04.379332    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:04.379427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:04.379398    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:04.379462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:04.379427    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:05.315152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.315108    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" event={"ID":"36eb9dc3-6c57-42b7-850e-72c689bb3a05","Type":"ContainerStarted","Data":"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"}
Oct 30 22:05:05.315558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.315168    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" event={"ID":"36eb9dc3-6c57-42b7-850e-72c689bb3a05","Type":"ContainerStarted","Data":"a555588a58ee2862bdab44a688d01b87c6ec75afc035a66f2e6d117e3fb8bdc9"}
Oct 30 22:05:05.471350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.471311    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:05.484740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.484688    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podStartSLOduration=2.484668619 podStartE2EDuration="2.484668619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:05.481977825 +0000 UTC m=+1358.570715640" watchObservedRunningTime="2026-10-30 22:05:05.484668619 +0000 UTC m=+1358.573406421"
Oct 30 22:05:05.962799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.959061    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:05.970811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.969866    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:05.970811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:05.969921    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:06.965152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:06.965119    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:06.965745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:06.965712    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:07.961798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:07.949078    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:07.961798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:07.949135    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:08.958798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:08.952598    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:08.958798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:08.952659    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:09.951806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:09.949107    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:09.951806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:09.949170    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:10.390212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.390172    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:10.663447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.663363    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:10.858115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.858078    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"]
Oct 30 22:05:10.863220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.863198    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:10.885557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.885529    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:10.915611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.915540    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:10.948249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.948203    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:05:10.948423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.948270    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 22:05:10.965092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.965067    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:10.965237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.965091    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:05:10.965237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.965202    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:10.971465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.971441    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"]
Oct 30 22:05:10.993516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.993485    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:10.993639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.993558    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:10.993697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:10.993640    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqbp\" (UniqueName: \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.094757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.094716    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.094960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.094859    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.094960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.094898    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqbp\" (UniqueName: \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.095575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.095548    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.097402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.097370    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.154998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.154962    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqbp\" (UniqueName: \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\") pod \"maas-api-revoke-sub-6d7c2c75-vr5tj\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.179216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.179153    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:11.567199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.567166    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"]
Oct 30 22:05:11.567615 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:05:11.567492    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccbab94_cd25_4042_a961_081d13529e9f.slice/crio-26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4 WatchSource:0}: Error finding container 26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4: Status 404 returned error can't find the container with id 26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4
Oct 30 22:05:11.951703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.951674    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:11.951979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.951960    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:11.952591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:11.952571    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:05:12.372197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:12.372159    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj" event={"ID":"4ccbab94-cd25-4042-a961-081d13529e9f","Type":"ContainerStarted","Data":"f7f9f7b4cc4d0ac1d598d17ae4c7154fd8436354eae6fe362e0c348fbea60269"}
Oct 30 22:05:12.372371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:12.372224    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj" event={"ID":"4ccbab94-cd25-4042-a961-081d13529e9f","Type":"ContainerStarted","Data":"26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4"}
Oct 30 22:05:12.489283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:12.489217    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj" podStartSLOduration=2.489197425 podStartE2EDuration="2.489197425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:12.4854544 +0000 UTC m=+1365.574192217" watchObservedRunningTime="2026-10-30 22:05:12.489197425 +0000 UTC m=+1365.577935230"
Oct 30 22:05:13.381310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:13.381272    4410 generic.go:356] "Generic (PLEG): container finished" podID="4ccbab94-cd25-4042-a961-081d13529e9f" containerID="f7f9f7b4cc4d0ac1d598d17ae4c7154fd8436354eae6fe362e0c348fbea60269" exitCode=0
Oct 30 22:05:13.381952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:13.381323    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj" event={"ID":"4ccbab94-cd25-4042-a961-081d13529e9f","Type":"ContainerDied","Data":"f7f9f7b4cc4d0ac1d598d17ae4c7154fd8436354eae6fe362e0c348fbea60269"}
Oct 30 22:05:14.535189 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.535166    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:14.643387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.643355    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\") pod \"4ccbab94-cd25-4042-a961-081d13529e9f\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") "
Oct 30 22:05:14.643546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.643508    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\" (UniqueName: \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\") pod \"4ccbab94-cd25-4042-a961-081d13529e9f\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") "
Oct 30 22:05:14.643649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.643633    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\") pod \"4ccbab94-cd25-4042-a961-081d13529e9f\" (UID: \"4ccbab94-cd25-4042-a961-081d13529e9f\") "
Oct 30 22:05:14.643736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.643708    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca" pod "4ccbab94-cd25-4042-a961-081d13529e9f" (UID: "4ccbab94-cd25-4042-a961-081d13529e9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:14.644109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.644085    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.645523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.645501    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca" pod "4ccbab94-cd25-4042-a961-081d13529e9f" (UID: "4ccbab94-cd25-4042-a961-081d13529e9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:14.645673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.645651    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp" pod "4ccbab94-cd25-4042-a961-081d13529e9f" (UID: "4ccbab94-cd25-4042-a961-081d13529e9f"). InnerVolumeSpecName "kube-api-access-csqbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:14.745299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.745268    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqbp\" (UniqueName: \"kubernetes.io/projected/4ccbab94-cd25-4042-a961-081d13529e9f-kube-api-access-csqbp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.745299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:14.745297    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbab94-cd25-4042-a961-081d13529e9f-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:15.395352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:15.395320    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"
Oct 30 22:05:15.395352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:15.395340    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj" event={"ID":"4ccbab94-cd25-4042-a961-081d13529e9f","Type":"ContainerDied","Data":"26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4"}
Oct 30 22:05:15.395543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:15.395389    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b0d9e28258571bf6de0591b388ea4820e319b5e6aade7c25e14fcc71f08ca4"
Oct 30 22:05:15.568199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:15.568168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:19.559798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.552256    4410 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.632798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.631848    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 22:05:19.691798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.680867    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-r6nlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:19.691798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.680933    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.821179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.821083    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.822592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.822557    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.842874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.842839    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.844820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:19.843822    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.026833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.016471    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.041805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.034875    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.041805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.034926    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.042054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.034948    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.060083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.060042    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.162427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.162384    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.662983 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:05:20.662943    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:05:20.886797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.885847    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:20.886797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:20.885909    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:21.365825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:21.365740    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:21.815076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:21.815030    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:26.965394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:26.965360    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:27.305050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.304986    4410 scope.go:122] "RemoveContainer" containerID="ab06169b6dec4a20b374494feadfdabd95426671e1e859c97220750fe4c07cf9"
Oct 30 22:05:27.314204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.314181    4410 scope.go:122] "RemoveContainer" containerID="61ddac12c296cf274af6ae317ac072ca74c0ba982604631c680a07524512f73c"
Oct 30 22:05:27.322628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.322608    4410 scope.go:122] "RemoveContainer" containerID="d9174e240318b5600ec8baf256fbec3e786da43fd2f4ed3fd2311d0aed1285bc"
Oct 30 22:05:27.333174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.333151    4410 scope.go:122] "RemoveContainer" containerID="f73a46e56a85a719402d5443813799a4df3909f86c0f8efdf93fda2cc3160219"
Oct 30 22:05:27.338325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.338299    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"]
Oct 30 22:05:27.338844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.338830    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4ccbab94-cd25-4042-a961-081d13529e9f" containerName="revoke-keys"
Oct 30 22:05:27.338893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.338845    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccbab94-cd25-4042-a961-081d13529e9f" containerName="revoke-keys"
Oct 30 22:05:27.338939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.338931    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccbab94-cd25-4042-a961-081d13529e9f" containerName="revoke-keys"
Oct 30 22:05:27.343465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.343445    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.397866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.397838    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfpg\" (UniqueName: \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.398018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.397997    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.398098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.398055    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.401754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.401732    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"]
Oct 30 22:05:27.446577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.446558    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:27.446682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.446571    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:27.446682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.446582    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:27.446682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.446631    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:05:27.499418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.499391    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfpg\" (UniqueName: \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.499550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.499506    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.499614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.499553    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.500328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.500302    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.502221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.502199    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.557036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.556973    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfpg\" (UniqueName: \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\") pod \"maas-api-revoke-sub-89b53197-rr7xh\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:27.657860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:27.657834    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:28.038870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:28.038841    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"]
Oct 30 22:05:28.041406 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:05:28.041378    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0154b759_6057_475e_8eff_644e71027e8e.slice/crio-d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775 WatchSource:0}: Error finding container d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775: Status 404 returned error can't find the container with id d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775
Oct 30 22:05:28.053954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:28.053928    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:28.518123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:28.518083    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh" event={"ID":"0154b759-6057-475e-8eff-644e71027e8e","Type":"ContainerStarted","Data":"062f78f0ffd6cfcaf958984a6516d5d41c21c5546db291fd5f3745ac66965cad"}
Oct 30 22:05:28.518290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:28.518145    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh" event={"ID":"0154b759-6057-475e-8eff-644e71027e8e","Type":"ContainerStarted","Data":"d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775"}
Oct 30 22:05:28.634113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:28.634022    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh" podStartSLOduration=1.634007014 podStartE2EDuration="1.634007014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:28.632529014 +0000 UTC m=+1381.721266818" watchObservedRunningTime="2026-10-30 22:05:28.634007014 +0000 UTC m=+1381.722744848"
Oct 30 22:05:29.526302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:29.526267    4410 generic.go:356] "Generic (PLEG): container finished" podID="0154b759-6057-475e-8eff-644e71027e8e" containerID="062f78f0ffd6cfcaf958984a6516d5d41c21c5546db291fd5f3745ac66965cad" exitCode=0
Oct 30 22:05:29.526683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:29.526357    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh" event={"ID":"0154b759-6057-475e-8eff-644e71027e8e","Type":"ContainerDied","Data":"062f78f0ffd6cfcaf958984a6516d5d41c21c5546db291fd5f3745ac66965cad"}
Oct 30 22:05:30.688540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.688512    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:30.740226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.740199    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\") pod \"0154b759-6057-475e-8eff-644e71027e8e\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") "
Oct 30 22:05:30.740354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.740339    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\" (UniqueName: \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\") pod \"0154b759-6057-475e-8eff-644e71027e8e\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") "
Oct 30 22:05:30.740398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.740374    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\") pod \"0154b759-6057-475e-8eff-644e71027e8e\" (UID: \"0154b759-6057-475e-8eff-644e71027e8e\") "
Oct 30 22:05:30.740537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.740517    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca" pod "0154b759-6057-475e-8eff-644e71027e8e" (UID: "0154b759-6057-475e-8eff-644e71027e8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:30.741025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.741005    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0154b759-6057-475e-8eff-644e71027e8e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.742445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.742424    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg" pod "0154b759-6057-475e-8eff-644e71027e8e" (UID: "0154b759-6057-475e-8eff-644e71027e8e"). InnerVolumeSpecName "kube-api-access-zxfpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:30.742445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.742418    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca" pod "0154b759-6057-475e-8eff-644e71027e8e" (UID: "0154b759-6057-475e-8eff-644e71027e8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:30.842494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.842400    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfpg\" (UniqueName: \"kubernetes.io/projected/0154b759-6057-475e-8eff-644e71027e8e-kube-api-access-zxfpg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.842494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:30.842428    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0154b759-6057-475e-8eff-644e71027e8e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:31.539194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:31.539162    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"
Oct 30 22:05:31.539404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:31.539189    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh" event={"ID":"0154b759-6057-475e-8eff-644e71027e8e","Type":"ContainerDied","Data":"d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775"}
Oct 30 22:05:31.539404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:31.539239    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a7981b544fe2dd0cfe10d8618573417a5006861b058d450f6e7aefcd57e775"
Oct 30 22:05:31.856086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:31.855999    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:33.033201 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:33.033163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:34.226886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:34.226853    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.849901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:36.849869    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:38.394665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:38.394616    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:39.055763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:39.055566    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:39.543527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:39.543473    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:41.964793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:41.964748    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:43.561993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:43.561956    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:44.169599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:44.169565    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:44.812878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:44.812840    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:46.943128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:46.943090    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:50.522123 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:05:50.522090    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:05:52.043871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:52.043833    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:53.654403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:53.654369    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:53.838373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:53.838327    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:54.007374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:54.007265    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:56.857954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:56.857905    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:57.515106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:57.515053    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:57.677098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:57.677064    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:57.844167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:57.844083    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.166118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:59.166073    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.365656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:59.365602    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.539431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:05:59.539354    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:02.609326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:02.609288    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:02.774536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:02.774502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:03.004974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.004942    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:06:03.005719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.005696    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0154b759-6057-475e-8eff-644e71027e8e" containerName="revoke-keys"
Oct 30 22:06:03.005719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.005717    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0154b759-6057-475e-8eff-644e71027e8e" containerName="revoke-keys"
Oct 30 22:06:03.005928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.005873    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0154b759-6057-475e-8eff-644e71027e8e" containerName="revoke-keys"
Oct 30 22:06:03.009901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.009880    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.033169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.033138    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:03.070847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.070813    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:06:03.097494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.097455    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.097681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.097522    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtkj\" (UniqueName: \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.114491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.114457    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-da0f3d-dockercfg-xgrxt\""
Oct 30 22:06:03.114672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.114561    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-da0f3d\""
Oct 30 22:06:03.198542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.198505    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.198753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.198558    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtkj\" (UniqueName: \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.199480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.199449    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.255686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.255599    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtkj\" (UniqueName: \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\") pod \"payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.332449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.332414    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:03.710101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.710068    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:06:03.710827 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:06:03.710801    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb621b3b_e125_4dbd_bd20_6331d7599779.slice/crio-776904a49d284c6065be14d089bce1307adc621dedea01214c524c633cee69cd WatchSource:0}: Error finding container 776904a49d284c6065be14d089bce1307adc621dedea01214c524c633cee69cd: Status 404 returned error can't find the container with id 776904a49d284c6065be14d089bce1307adc621dedea01214c524c633cee69cd
Oct 30 22:06:03.712651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.712636    4410 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:03.765643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:03.765615    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" event={"ID":"cb621b3b-e125-4dbd-bd20-6331d7599779","Type":"ContainerStarted","Data":"776904a49d284c6065be14d089bce1307adc621dedea01214c524c633cee69cd"}
Oct 30 22:06:06.792575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:06.792537    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" event={"ID":"cb621b3b-e125-4dbd-bd20-6331d7599779","Type":"ContainerStarted","Data":"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"}
Oct 30 22:06:06.793059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:06.792671    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:06.904811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:06.904739    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" podStartSLOduration=2.321911762 podStartE2EDuration="4.904723062s" totalImagesPullingTime="2.5828113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:06.904299165 +0000 UTC m=+1419.993036969" watchObservedRunningTime="2026-10-30 22:06:06.904723062 +0000 UTC m=+1419.993460865"
Oct 30 22:06:08.451323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:08.451280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.451435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:13.451394    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:17.803625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:17.803595    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:06:19.205059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.205026    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:06:19.208991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.208970    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.286278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.286228    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:19.309890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.309861    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.309917    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310057    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310173    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310236    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310318    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310492    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310621    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxtm\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.310746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.310658    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.320250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.320203    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:06:19.411815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.411751    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.411970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.411840    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxtm\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.411970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.411877    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.411985    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412036    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412096    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412167    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412237    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412307    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412510    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412570    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412590    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412671    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.412713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.412667    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.414506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.414478    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.414589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.414571    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.468867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.468810    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.468958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.468944    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxtm\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.523632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.523607    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:19.953066 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:06:19.953036    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3420b78_7732_4efa_9fee_a3927a67b983.slice/crio-16a64adda670568226ce39c98aa5291d849174d76ffd10d4a862ab56b31124b9 WatchSource:0}: Error finding container 16a64adda670568226ce39c98aa5291d849174d76ffd10d4a862ab56b31124b9: Status 404 returned error can't find the container with id 16a64adda670568226ce39c98aa5291d849174d76ffd10d4a862ab56b31124b9
Oct 30 22:06:19.953285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.953263    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:06:19.955446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.955412    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:19.955570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.955503    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:19.955570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:19.955561    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:20.523402 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:06:20.523358    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:06:20.583565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.583528    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:06:20.587663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.587639    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634491    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634525    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634558    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634701    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634733    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.634834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.634822    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p48c\" (UniqueName: \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.689409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.689374    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7fabe91d\"/\"openshift-service-ca.crt\""
Oct 30 22:06:20.689838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.689814    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7fabe91d\"/\"default-dockercfg-wlpkt\""
Oct 30 22:06:20.689972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.689844    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7fabe91d\"/\"kube-root-ca.crt\""
Oct 30 22:06:20.689972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.689855    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7fabe91d\"/\"e2e-lifecycle-model-7fabe91d-kserve-self-signed-certs\""
Oct 30 22:06:20.694729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.694704    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:06:20.708251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.708223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:20.736035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.735999    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.736526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.736502    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.736724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.736707    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p48c\" (UniqueName: \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.737378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.737354    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.737493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.737391    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.737493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.737430    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.738020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.737996    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.740753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.739446    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.740753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.739686    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.743862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.743721    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.785490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.785403    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.805496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.803128    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p48c\" (UniqueName: \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\") pod \"e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") " pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:20.892853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.892815    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" event={"ID":"b3420b78-7732-4efa-9fee-a3927a67b983","Type":"ContainerStarted","Data":"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"}
Oct 30 22:06:20.893043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.892879    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" event={"ID":"b3420b78-7732-4efa-9fee-a3927a67b983","Type":"ContainerStarted","Data":"16a64adda670568226ce39c98aa5291d849174d76ffd10d4a862ab56b31124b9"}
Oct 30 22:06:20.901276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:20.901252    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:21.021236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.021182    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podStartSLOduration=2.02116141 podStartE2EDuration="2.02116141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:21.016598568 +0000 UTC m=+1434.105336371" watchObservedRunningTime="2026-10-30 22:06:21.02116141 +0000 UTC m=+1434.109899214"
Oct 30 22:06:21.324742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.324704    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:06:21.325070 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:06:21.325039    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18145f3b_09f3_4a7e_9c2e_60ea8802742f.slice/crio-77055e02cc192214637440db9f26787afd4488558889d52c6e49147412ae78a6 WatchSource:0}: Error finding container 77055e02cc192214637440db9f26787afd4488558889d52c6e49147412ae78a6: Status 404 returned error can't find the container with id 77055e02cc192214637440db9f26787afd4488558889d52c6e49147412ae78a6
Oct 30 22:06:21.523955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.523923    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:21.525299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.525273    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:21.525419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.525320    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:21.726558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.726519    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:21.901356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.901317    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" event={"ID":"18145f3b-09f3-4a7e-9c2e-60ea8802742f","Type":"ContainerStarted","Data":"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"}
Oct 30 22:06:21.901536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.901363    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" event={"ID":"18145f3b-09f3-4a7e-9c2e-60ea8802742f","Type":"ContainerStarted","Data":"77055e02cc192214637440db9f26787afd4488558889d52c6e49147412ae78a6"}
Oct 30 22:06:21.901701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:21.901686    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:22.026492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:22.026435    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" podStartSLOduration=2.026415157 podStartE2EDuration="2.026415157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:22.024211904 +0000 UTC m=+1435.112949732" watchObservedRunningTime="2026-10-30 22:06:22.026415157 +0000 UTC m=+1435.115152965"
Oct 30 22:06:22.524849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:22.524814    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:22.525240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:22.524870    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:23.163795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:23.155296    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:23.385052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:23.385017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:23.524032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:23.523937    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:23.524032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:23.524011    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:24.524124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:24.524087    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:24.524499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:24.524147    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:25.524100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:25.524057    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:25.524284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:25.524127    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:26.524899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:26.524863    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 22:06:26.525313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:26.524927    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 22:06:27.528594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:27.528562    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:27.529070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:27.528900    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:27.529866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:27.529845    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:06:27.898504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:27.898472    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:28.097117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:28.097083    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:28.246696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:28.246615    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:30.902290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:30.902249    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:30.915169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:30.915146    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:30.973500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:30.973475    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:06:32.700814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.700768    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"]
Oct 30 22:06:32.705073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.705050    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.761613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.761586    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"]
Oct 30 22:06:32.796797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.796755    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:32.796980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.796763    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:06:32.796980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.796769    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:32.805517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.805500    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:32.813439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.813417    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.813535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.813482    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.813601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.813584    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp4x\" (UniqueName: \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.915392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.915357    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.915524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.915433    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.915524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.915507    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp4x\" (UniqueName: \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.916168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.916143    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.917754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.917735    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:32.952186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.952133    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:32.972171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:32.972145    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp4x\" (UniqueName: \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\") pod \"maas-api-revoke-sub-2ba22b57-fsn9s\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:33.018534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:33.018504    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:33.406836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:33.406808    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"]
Oct 30 22:06:33.407458 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:06:33.407435    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5324f5a4_40f5_4caf_85e4_2f2a12546b48.slice/crio-a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5 WatchSource:0}: Error finding container a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5: Status 404 returned error can't find the container with id a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5
Oct 30 22:06:33.984609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:33.984520    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s" event={"ID":"5324f5a4-40f5-4caf-85e4-2f2a12546b48","Type":"ContainerStarted","Data":"586f8939f58e28b317430b1e9721eb49e990a2778a0b9dec90f017683ba08e6c"}
Oct 30 22:06:33.984609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:33.984577    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s" event={"ID":"5324f5a4-40f5-4caf-85e4-2f2a12546b48","Type":"ContainerStarted","Data":"a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5"}
Oct 30 22:06:34.103043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:34.102996    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s" podStartSLOduration=2.102983419 podStartE2EDuration="2.102983419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:34.101448113 +0000 UTC m=+1447.190185916" watchObservedRunningTime="2026-10-30 22:06:34.102983419 +0000 UTC m=+1447.191721222"
Oct 30 22:06:34.253556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:34.253499    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:34.993065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:34.993031    4410 generic.go:356] "Generic (PLEG): container finished" podID="5324f5a4-40f5-4caf-85e4-2f2a12546b48" containerID="586f8939f58e28b317430b1e9721eb49e990a2778a0b9dec90f017683ba08e6c" exitCode=0
Oct 30 22:06:34.993481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:34.993122    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s" event={"ID":"5324f5a4-40f5-4caf-85e4-2f2a12546b48","Type":"ContainerDied","Data":"586f8939f58e28b317430b1e9721eb49e990a2778a0b9dec90f017683ba08e6c"}
Oct 30 22:06:36.153392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.153365    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:36.257950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.257895    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\") pod \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") "
Oct 30 22:06:36.258060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.257947    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\" (UniqueName: \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\") pod \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") "
Oct 30 22:06:36.258106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.258062    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\") pod \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\" (UID: \"5324f5a4-40f5-4caf-85e4-2f2a12546b48\") "
Oct 30 22:06:36.258385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.258359    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca" pod "5324f5a4-40f5-4caf-85e4-2f2a12546b48" (UID: "5324f5a4-40f5-4caf-85e4-2f2a12546b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:36.258567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.258547    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:06:36.259966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.259945    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca" pod "5324f5a4-40f5-4caf-85e4-2f2a12546b48" (UID: "5324f5a4-40f5-4caf-85e4-2f2a12546b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:36.260042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.260025    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x" pod "5324f5a4-40f5-4caf-85e4-2f2a12546b48" (UID: "5324f5a4-40f5-4caf-85e4-2f2a12546b48"). InnerVolumeSpecName "kube-api-access-4fp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:36.359948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.359913    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5324f5a4-40f5-4caf-85e4-2f2a12546b48-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:06:36.359948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:36.359944    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fp4x\" (UniqueName: \"kubernetes.io/projected/5324f5a4-40f5-4caf-85e4-2f2a12546b48-kube-api-access-4fp4x\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:06:37.007443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:37.007409    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s" event={"ID":"5324f5a4-40f5-4caf-85e4-2f2a12546b48","Type":"ContainerDied","Data":"a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5"}
Oct 30 22:06:37.007443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:37.007442    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"
Oct 30 22:06:37.007650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:37.007480    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4be2e7b5cd5f80c20d580ed31c831ad82d2c4154c2dbec7f9c09b4273b36af5"
Oct 30 22:06:37.807039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:37.807008    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:38.002612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:38.002576    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:43.097695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:43.096443    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:48.167150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:48.167116    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.521747 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:06:50.521703    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:06:52.871742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:52.871701    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:57.691124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:57.691084    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:58.452755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:58.452715    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.124911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.124874    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:06:59.126234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.126206    4410 cpu_manager.go:414] "containerMap: removing container" podUID="5324f5a4-40f5-4caf-85e4-2f2a12546b48" containerName="revoke-keys"
Oct 30 22:06:59.126406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.126392    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5324f5a4-40f5-4caf-85e4-2f2a12546b48" containerName="revoke-keys"
Oct 30 22:06:59.126713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.126689    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="5324f5a4-40f5-4caf-85e4-2f2a12546b48" containerName="revoke-keys"
Oct 30 22:06:59.132034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.131992    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.167725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.167682    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:06:59.182421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.182386    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.211050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.211023    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d73ad7-openshift-default-dockercfg-flwbm\""
Oct 30 22:06:59.267825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.267784    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.267881    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268009    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268050    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv28t\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268175    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268240    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268279    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268374    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.268444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.268405    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.369705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.369667    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.369905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.369808    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370134    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv28t\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370207    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370281    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370295    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370365    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370398    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370471    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370499    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370552    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.370719    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.371154    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.371882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.371205    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.377446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.377379    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.385496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.385473    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.427698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.427661    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv28t\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.427963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.427937    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\") pod \"e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") " pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.471628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.471598    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:06:59.764427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.764388    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.910316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.910276    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:06:59.945572 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:06:59.945537    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dc1069_d7d7_4f20_a895_54c4ebac2716.slice/crio-505ff1ffd8c0f63f8157014b6b3e93901437056883e38c00f0c9b81d1a234860 WatchSource:0}: Error finding container 505ff1ffd8c0f63f8157014b6b3e93901437056883e38c00f0c9b81d1a234860: Status 404 returned error can't find the container with id 505ff1ffd8c0f63f8157014b6b3e93901437056883e38c00f0c9b81d1a234860
Oct 30 22:06:59.948348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.948301    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:59.948450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.948404    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:59.948496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:06:59.948448    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:00.195072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:00.195034    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerStarted","Data":"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"}
Oct 30 22:07:00.195452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:00.195099    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerStarted","Data":"505ff1ffd8c0f63f8157014b6b3e93901437056883e38c00f0c9b81d1a234860"}
Oct 30 22:07:00.317746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:00.317682    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podStartSLOduration=2.31766394 podStartE2EDuration="2.31766394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:00.311204857 +0000 UTC m=+1473.399942662" watchObservedRunningTime="2026-10-30 22:07:00.31766394 +0000 UTC m=+1473.406401746"
Oct 30 22:07:00.472048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:00.471964    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:07:01.477686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:01.477661    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:07:02.207476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:02.207447    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:07:02.208305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:02.208279    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:07:02.658632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:02.658601    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.563881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.563847    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:07:04.568072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.568054    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.613203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.613179    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:07:04.616986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.616968    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.723569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.723540    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:07:04.747578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.747549    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.747739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.747612    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.747804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.747729    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.747865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.747809    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.747968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.747941    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748019    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748048    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748172    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748211    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748239    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748303    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748399    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748454    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhn7\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748491    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748527    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.748590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748553    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748586    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx55\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.748666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.748630    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.771996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.771973    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:07:04.849598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849519    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.849598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849572    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.849856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849607    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.849856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849631    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.849856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849667    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.849856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849725    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.849856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849746    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849864    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849896    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849919    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849946    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.849978    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850023    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhn7\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850056    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850093    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850115    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850120    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850165    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx55\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850216    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.850660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850372    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850711    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.850862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850735    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.851011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.850988    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.851075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.851056    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.851134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.851086    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.851134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.851088    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.851360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.851335    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.851612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.851583    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.853893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.853865    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.854241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.854217    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.854321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.854296    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.854501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.854476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.906018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.905991    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.907164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.907142    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhn7\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.907255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.907167    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx55\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:04.907979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.907958    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:04.929958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:04.929939    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:05.181952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.181922    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:05.361910 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:05.361881    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6785ae40_2209_4747_8b0d_387c81c481b2.slice/crio-03ce133331fe1af8496b12104019e478633831b63b25e93b903663feaa680eac WatchSource:0}: Error finding container 03ce133331fe1af8496b12104019e478633831b63b25e93b903663feaa680eac: Status 404 returned error can't find the container with id 03ce133331fe1af8496b12104019e478633831b63b25e93b903663feaa680eac
Oct 30 22:07:05.362108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.362085    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:07:05.364279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.364251    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:05.364339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.364322    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:05.364374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.364350    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:05.565332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.565297    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:07:05.565714 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:05.565681    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e6b31d_a8c1_4e62_81a9_d62d14c44932.slice/crio-3393aed454e863b770f1b738091c9301a1aa4d8375d17c470472332c126021cd WatchSource:0}: Error finding container 3393aed454e863b770f1b738091c9301a1aa4d8375d17c470472332c126021cd: Status 404 returned error can't find the container with id 3393aed454e863b770f1b738091c9301a1aa4d8375d17c470472332c126021cd
Oct 30 22:07:05.568182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.568151    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:05.568281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.568212    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:05.568281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:05.568241    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:06.241160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.241111    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" event={"ID":"92e6b31d-a8c1-4e62-81a9-d62d14c44932","Type":"ContainerStarted","Data":"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"}
Oct 30 22:07:06.241374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.241185    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" event={"ID":"92e6b31d-a8c1-4e62-81a9-d62d14c44932","Type":"ContainerStarted","Data":"3393aed454e863b770f1b738091c9301a1aa4d8375d17c470472332c126021cd"}
Oct 30 22:07:06.243060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.243039    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" event={"ID":"6785ae40-2209-4747-8b0d-387c81c481b2","Type":"ContainerStarted","Data":"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"}
Oct 30 22:07:06.243176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.243077    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" event={"ID":"6785ae40-2209-4747-8b0d-387c81c481b2","Type":"ContainerStarted","Data":"03ce133331fe1af8496b12104019e478633831b63b25e93b903663feaa680eac"}
Oct 30 22:07:06.469797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.468217    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podStartSLOduration=2.468198356 podStartE2EDuration="2.468198356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:06.408236645 +0000 UTC m=+1479.496974448" watchObservedRunningTime="2026-10-30 22:07:06.468198356 +0000 UTC m=+1479.556936161"
Oct 30 22:07:06.537990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.537888    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podStartSLOduration=2.53787385 podStartE2EDuration="2.53787385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:06.534865708 +0000 UTC m=+1479.623603511" watchObservedRunningTime="2026-10-30 22:07:06.53787385 +0000 UTC m=+1479.626611653"
Oct 30 22:07:06.930559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.930525    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:06.931580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.931555    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:06.931666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:06.931607    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:07.184855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.184465    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:07.184855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.184597    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:07.184855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.184642    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:07.597345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.597245    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.930205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.930173    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:07.930504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:07.930467    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:08.183239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:08.183147    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:08.183869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:08.183832    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:08.930483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:08.930450    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:08.930658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:08.930517    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:09.185857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:09.185573    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.185857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:09.185752    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:09.931119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:09.931033    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.931119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:09.931090    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:10.184953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:10.184437    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.184953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:10.184500    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:10.931076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:10.931040    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.931481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:10.931103    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:11.186678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:11.186583    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.186994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:11.186953    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:11.930878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:11.930823    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.931094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:11.930909    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:07:12.182399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:12.182237    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:07:12.182399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:12.182306    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:07:13.195762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.195725    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:13.373015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.372982    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:13.374503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.374478    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:13.379714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.379689    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:07:13.393033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.393003    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:13.393931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.393906    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:13.394946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.394921    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:07:13.416043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:13.416007    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.588296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:16.588249    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.790008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:16.789967    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.237248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.237215    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"]
Oct 30 22:07:17.242652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.242624    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.330714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.330681    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:17.344789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.342087    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"]
Oct 30 22:07:17.345983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.345950    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.346136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.346124    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bqw\" (UniqueName: \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.346340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.346319    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.349816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.347206    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:07:17.349816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.347205    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:17.349816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.347206    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:17.447567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.447532    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.447738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.447686    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bqw\" (UniqueName: \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.447852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.447837    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.448415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.448390    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.450306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.450279    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.504746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.504689    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bqw\" (UniqueName: \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\") pod \"maas-api-revoke-sub-15815553-pfx42\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.561632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.561609    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:17.941939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:17.941910    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"]
Oct 30 22:07:17.942339 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:17.942311    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dc8093_53fa_4afe_bbb7_dfc95af8dd52.slice/crio-ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62 WatchSource:0}: Error finding container ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62: Status 404 returned error can't find the container with id ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62
Oct 30 22:07:18.395601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:18.395563    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42" event={"ID":"32dc8093-53fa-4afe-bbb7-dfc95af8dd52","Type":"ContainerStarted","Data":"7272e2674e2243a53fb559639c4003eb8e7226197160dc9414b51e42afed10aa"}
Oct 30 22:07:18.396089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:18.395612    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42" event={"ID":"32dc8093-53fa-4afe-bbb7-dfc95af8dd52","Type":"ContainerStarted","Data":"ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62"}
Oct 30 22:07:18.510369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:18.510271    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42" podStartSLOduration=1.5102569460000002 podStartE2EDuration="1.510256946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:18.508934183 +0000 UTC m=+1491.597672008" watchObservedRunningTime="2026-10-30 22:07:18.510256946 +0000 UTC m=+1491.598994745"
Oct 30 22:07:19.113387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:19.113305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:19.408945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:19.408906    4410 generic.go:356] "Generic (PLEG): container finished" podID="32dc8093-53fa-4afe-bbb7-dfc95af8dd52" containerID="7272e2674e2243a53fb559639c4003eb8e7226197160dc9414b51e42afed10aa" exitCode=0
Oct 30 22:07:19.409135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:19.408999    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42" event={"ID":"32dc8093-53fa-4afe-bbb7-dfc95af8dd52","Type":"ContainerDied","Data":"7272e2674e2243a53fb559639c4003eb8e7226197160dc9414b51e42afed10aa"}
Oct 30 22:07:19.772321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:19.772225    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.519460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.519421    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.526811 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:07:20.526765    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:07:20.570582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.570557    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:20.694366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.694279    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\" (UniqueName: \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\") pod \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") "
Oct 30 22:07:20.694532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.694439    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\") pod \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") "
Oct 30 22:07:20.694603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.694531    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\") pod \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\" (UID: \"32dc8093-53fa-4afe-bbb7-dfc95af8dd52\") "
Oct 30 22:07:20.695050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.695018    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca" pod "32dc8093-53fa-4afe-bbb7-dfc95af8dd52" (UID: "32dc8093-53fa-4afe-bbb7-dfc95af8dd52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:20.695493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.695466    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:20.696938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.696911    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw" pod "32dc8093-53fa-4afe-bbb7-dfc95af8dd52" (UID: "32dc8093-53fa-4afe-bbb7-dfc95af8dd52"). InnerVolumeSpecName "kube-api-access-g4bqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:20.709926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.709893    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca" pod "32dc8093-53fa-4afe-bbb7-dfc95af8dd52" (UID: "32dc8093-53fa-4afe-bbb7-dfc95af8dd52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:20.797287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.797259    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bqw\" (UniqueName: \"kubernetes.io/projected/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-kube-api-access-g4bqw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:20.797287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:20.797290    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dc8093-53fa-4afe-bbb7-dfc95af8dd52-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:21.206467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:21.190017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:21.428383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:21.428343    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"
Oct 30 22:07:21.428696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:21.428405    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42" event={"ID":"32dc8093-53fa-4afe-bbb7-dfc95af8dd52","Type":"ContainerDied","Data":"ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62"}
Oct 30 22:07:21.428696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:21.428464    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee19d8b57c6c0ed372db9a5002eeed7b9193b9b8f0c5d8826345e1a7d394ed62"
Oct 30 22:07:22.029497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:22.029444    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:22.246260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:22.246218    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:24.791119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:24.791074    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:27.048144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:27.048114    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"]
Oct 30 22:07:27.097122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:27.097084    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a6bccba-hs9z2"]
Oct 30 22:07:28.095695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.095651    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"]
Oct 30 22:07:28.096512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.096481    4410 cpu_manager.go:414] "containerMap: removing container" podUID="32dc8093-53fa-4afe-bbb7-dfc95af8dd52" containerName="revoke-keys"
Oct 30 22:07:28.096644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.096512    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dc8093-53fa-4afe-bbb7-dfc95af8dd52" containerName="revoke-keys"
Oct 30 22:07:28.096710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.096666    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dc8093-53fa-4afe-bbb7-dfc95af8dd52" containerName="revoke-keys"
Oct 30 22:07:28.100511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.100489    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.157482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.157460    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"]
Oct 30 22:07:28.197872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.197840    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnd4w\" (UniqueName: \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.198072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.198039    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.198163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.198125    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.201224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.201202    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:28.201326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.201236    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:28.201326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.201245    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:07:28.202354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.202341    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:28.299448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.299411    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnd4w\" (UniqueName: \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.299640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.299544    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.299640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.299580    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.300258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.300232    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.302091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.302062    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.356014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.355957    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnd4w\" (UniqueName: \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\") pod \"maas-api-revoke-sub-e2a5a1b8-9n79m\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.413525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.413501    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:28.788322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:28.788292    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"]
Oct 30 22:07:28.788668 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:28.788646    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7c7ac8_28b9_4fa9_8a83_135500a06b63.slice/crio-c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1 WatchSource:0}: Error finding container c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1: Status 404 returned error can't find the container with id c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1
Oct 30 22:07:29.021538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:29.021501    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170cf1f4-f583-4ba5-9eac-5d4848c48432" path="/var/lib/kubelet/pods/170cf1f4-f583-4ba5-9eac-5d4848c48432/volumes"
Oct 30 22:07:29.484690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:29.484650    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m" event={"ID":"4d7c7ac8-28b9-4fa9-8a83-135500a06b63","Type":"ContainerStarted","Data":"0c08584a083d7f508dedc8922e8f5b97f58087f055c3cc6c3485a979bd54c9d1"}
Oct 30 22:07:29.485184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:29.484708    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m" event={"ID":"4d7c7ac8-28b9-4fa9-8a83-135500a06b63","Type":"ContainerStarted","Data":"c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1"}
Oct 30 22:07:29.600549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:29.600493    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m" podStartSLOduration=1.6004792700000001 podStartE2EDuration="1.60047927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:29.596842716 +0000 UTC m=+1502.685580520" watchObservedRunningTime="2026-10-30 22:07:29.60047927 +0000 UTC m=+1502.689217072"
Oct 30 22:07:30.247509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:30.247456    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.493477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:30.493440    4410 generic.go:356] "Generic (PLEG): container finished" podID="4d7c7ac8-28b9-4fa9-8a83-135500a06b63" containerID="0c08584a083d7f508dedc8922e8f5b97f58087f055c3cc6c3485a979bd54c9d1" exitCode=0
Oct 30 22:07:30.493892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:30.493516    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m" event={"ID":"4d7c7ac8-28b9-4fa9-8a83-135500a06b63","Type":"ContainerDied","Data":"0c08584a083d7f508dedc8922e8f5b97f58087f055c3cc6c3485a979bd54c9d1"}
Oct 30 22:07:31.648955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.648931    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:31.741388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.741362    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\") pod \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") "
Oct 30 22:07:31.741572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.741556    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\") pod \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") "
Oct 30 22:07:31.741632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.741614    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\" (UniqueName: \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\") pod \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\" (UID: \"4d7c7ac8-28b9-4fa9-8a83-135500a06b63\") "
Oct 30 22:07:31.742004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.741975    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca" pod "4d7c7ac8-28b9-4fa9-8a83-135500a06b63" (UID: "4d7c7ac8-28b9-4fa9-8a83-135500a06b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:31.742196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.742176    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:31.743361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.743337    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca" pod "4d7c7ac8-28b9-4fa9-8a83-135500a06b63" (UID: "4d7c7ac8-28b9-4fa9-8a83-135500a06b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:31.743597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.743575    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w" pod "4d7c7ac8-28b9-4fa9-8a83-135500a06b63" (UID: "4d7c7ac8-28b9-4fa9-8a83-135500a06b63"). InnerVolumeSpecName "kube-api-access-fnd4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:31.843713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.843686    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnd4w\" (UniqueName: \"kubernetes.io/projected/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-kube-api-access-fnd4w\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:31.843713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:31.843711    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7c7ac8-28b9-4fa9-8a83-135500a06b63-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.507485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:32.507450    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"
Oct 30 22:07:32.507655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:32.507485    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m" event={"ID":"4d7c7ac8-28b9-4fa9-8a83-135500a06b63","Type":"ContainerDied","Data":"c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1"}
Oct 30 22:07:32.507655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:32.507540    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e573ad3ad91df409d5f04ca6a46b8e265321b41dfb1c60da7f3aeb1713fea1"
Oct 30 22:07:35.270519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:35.270485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:35.436729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:35.436692    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:35.643326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:35.643288    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:35.829426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:35.829389    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.221445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:36.221401    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.909625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:36.909576    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:37.314035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:37.313940    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:38.815797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:38.815377    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:39.127934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.112193    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:39.263948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.263912    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:07:39.282175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.282142    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4d7c7ac8-28b9-4fa9-8a83-135500a06b63" containerName="revoke-keys"
Oct 30 22:07:39.282418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.282397    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7c7ac8-28b9-4fa9-8a83-135500a06b63" containerName="revoke-keys"
Oct 30 22:07:39.282804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.282768    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7c7ac8-28b9-4fa9-8a83-135500a06b63" containerName="revoke-keys"
Oct 30 22:07:39.418796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.404211    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.452503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.450629    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:07:39.581798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.556971    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-02a8fa-dockercfg-cb8cq\""
Oct 30 22:07:39.581798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.557148    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-02a8fa\""
Oct 30 22:07:39.635330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.634181    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.635330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.634280    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rsw\" (UniqueName: \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.780852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.762265    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.781165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.772566    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.781425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.781403    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rsw\" (UniqueName: \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:39.918033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:39.918003    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rsw\" (UniqueName: \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\") pod \"payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:40.212796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.212494    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:40.618557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.618527    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:07:40.618971 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:40.618937    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d7fee0_5f25_405a_aa09_1f7d569c78b8.slice/crio-2fe39beedfbf51d9da7fdaeb973f278b92afca53fbffc31b7f9e4747a2982565 WatchSource:0}: Error finding container 2fe39beedfbf51d9da7fdaeb973f278b92afca53fbffc31b7f9e4747a2982565: Status 404 returned error can't find the container with id 2fe39beedfbf51d9da7fdaeb973f278b92afca53fbffc31b7f9e4747a2982565
Oct 30 22:07:40.779618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.779589    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" event={"ID":"64d7fee0-5f25-405a-aa09-1f7d569c78b8","Type":"ContainerStarted","Data":"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"}
Oct 30 22:07:40.779768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.779637    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" event={"ID":"64d7fee0-5f25-405a-aa09-1f7d569c78b8","Type":"ContainerStarted","Data":"2fe39beedfbf51d9da7fdaeb973f278b92afca53fbffc31b7f9e4747a2982565"}
Oct 30 22:07:40.779843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.779788    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:40.969604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:40.969540    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" podStartSLOduration=1.969522187 podStartE2EDuration="1.969522187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:40.966362067 +0000 UTC m=+1514.055099872" watchObservedRunningTime="2026-10-30 22:07:40.969522187 +0000 UTC m=+1514.058259991"
Oct 30 22:07:41.484466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:41.484423    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:41.666469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:41.666437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:41.817952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:41.817863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.049656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:44.049626    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"]
Oct 30 22:07:44.098341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:44.098308    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba78669-9v6bg"]
Oct 30 22:07:44.746043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:44.745999    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.918285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:44.918225    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:45.025570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.024193    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8" path="/var/lib/kubelet/pods/9aedfdd9-c8e8-4a4b-b7e2-90ef97d4eab8/volumes"
Oct 30 22:07:45.140878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.140843    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:07:45.145346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.145323    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.162424 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.162400    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8cm\" (UniqueName: \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.162842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.162697    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.207291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.207003    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:07:45.248233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.248196    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-479363\""
Oct 30 22:07:45.248431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.248321    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-479363-dockercfg-fzppz\""
Oct 30 22:07:45.265851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.264080    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.265851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.264206    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8cm\" (UniqueName: \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.266139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.266111    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.320858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.320765    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8cm\" (UniqueName: \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\") pod \"payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.459968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.459941    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:45.837683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:45.837368    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:07:45.837931 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:07:45.837900    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1bf09f_42c7_4382_9eef_070316319ab5.slice/crio-78b76aaa3922b9c74a8f08d44af66f217f0efaefb6e6b5d30d2fccb03422302f WatchSource:0}: Error finding container 78b76aaa3922b9c74a8f08d44af66f217f0efaefb6e6b5d30d2fccb03422302f: Status 404 returned error can't find the container with id 78b76aaa3922b9c74a8f08d44af66f217f0efaefb6e6b5d30d2fccb03422302f
Oct 30 22:07:46.753958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:46.753921    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:46.830418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:46.830354    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" event={"ID":"5f1bf09f-42c7-4382-9eef-070316319ab5","Type":"ContainerStarted","Data":"25a0b165baa76471724d28dfa1b74457496ef1c502e75d59986d72565d3531d9"}
Oct 30 22:07:46.830802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:46.830742    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:07:46.830802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:46.830799    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" event={"ID":"5f1bf09f-42c7-4382-9eef-070316319ab5","Type":"ContainerStarted","Data":"78b76aaa3922b9c74a8f08d44af66f217f0efaefb6e6b5d30d2fccb03422302f"}
Oct 30 22:07:46.945063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:46.945011    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" podStartSLOduration=1.944997664 podStartE2EDuration="1.944997664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:46.943842851 +0000 UTC m=+1520.032580655" watchObservedRunningTime="2026-10-30 22:07:46.944997664 +0000 UTC m=+1520.033735467"
Oct 30 22:07:50.522893 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:07:50.522854    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:07:51.641531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:51.641463    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:51.791792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:51.791749    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:07:56.478806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:56.478736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:57.840265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:07:57.840223    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:08:00.049808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:00.049739    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"]
Oct 30 22:08:00.099461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:00.099432    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"]
Oct 30 22:08:00.103266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:00.103243    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984ae73-jj5pb"]
Oct 30 22:08:00.106407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:00.106387    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b36e14-jmm7t"]
Oct 30 22:08:01.020425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:01.020391    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d73aa03-f578-42b3-bed3-9ff90ee16d1a" path="/var/lib/kubelet/pods/6d73aa03-f578-42b3-bed3-9ff90ee16d1a/volumes"
Oct 30 22:08:01.020913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:01.020896    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4ba892-6d80-4d0a-91e9-862dca504d8a" path="/var/lib/kubelet/pods/7a4ba892-6d80-4d0a-91e9-862dca504d8a/volumes"
Oct 30 22:08:01.182570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:01.182536    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:01.403565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:01.403530    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:06.331094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:06.331061    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:11.049532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:11.049497    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"]
Oct 30 22:08:11.098650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:11.098621    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dffa1af5-98bxh"]
Oct 30 22:08:11.183533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:11.183502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:13.020490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:13.020447    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce88a89b-482e-4230-8861-fceda7ac3214" path="/var/lib/kubelet/pods/ce88a89b-482e-4230-8861-fceda7ac3214/volumes"
Oct 30 22:08:16.089052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:16.089017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:18.064238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:18.064196    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"]
Oct 30 22:08:18.113289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:18.113260    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53af5867-8g4w9"]
Oct 30 22:08:19.021446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:19.021414    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f308a692-491a-444b-a52e-9c4755d6c9c1" path="/var/lib/kubelet/pods/f308a692-491a-444b-a52e-9c4755d6c9c1/volumes"
Oct 30 22:08:20.163798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.161296    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"]
Oct 30 22:08:20.174228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.174202    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.206757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.206731    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"]
Oct 30 22:08:20.227079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.227051    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:20.270115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.270086    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:20.270115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.270104    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:08:20.270115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.270090    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:20.328319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.328288    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn5c\" (UniqueName: \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.328459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.328421    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.328611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.328595    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.430384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.430306    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.430384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.430374    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn5c\" (UniqueName: \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.430597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.430446    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.431027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.431001    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.432799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.432766    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.487051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.487021    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn5c\" (UniqueName: \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\") pod \"maas-api-revoke-sub-10b9da98-2rtqg\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.487216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.487198    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:20.527525 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:20.527484    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:08:20.900560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:20.900525    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"]
Oct 30 22:08:20.901471 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:08:20.901439    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9e0f75_9fd0_40a0_8875_d79524eb395f.slice/crio-9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86 WatchSource:0}: Error finding container 9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86: Status 404 returned error can't find the container with id 9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86
Oct 30 22:08:21.093224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:21.093137    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg" event={"ID":"ff9e0f75-9fd0-40a0-8875-d79524eb395f","Type":"ContainerStarted","Data":"9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86"}
Oct 30 22:08:21.108678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:21.108630    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:22.101665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:22.101632    4410 generic.go:356] "Generic (PLEG): container finished" podID="ff9e0f75-9fd0-40a0-8875-d79524eb395f" containerID="d46de72842dc3c9af196054c932f8d0e40012ee28b58922820d33a22c104b4f7" exitCode=0
Oct 30 22:08:22.102053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:22.101714    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg" event={"ID":"ff9e0f75-9fd0-40a0-8875-d79524eb395f","Type":"ContainerDied","Data":"d46de72842dc3c9af196054c932f8d0e40012ee28b58922820d33a22c104b4f7"}
Oct 30 22:08:22.472416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:22.472373    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.307612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.307587    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:23.394329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.394301    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\" (UniqueName: \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\") pod \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") "
Oct 30 22:08:23.394507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.394337    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\") pod \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") "
Oct 30 22:08:23.394720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.394696    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\") pod \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\" (UID: \"ff9e0f75-9fd0-40a0-8875-d79524eb395f\") "
Oct 30 22:08:23.395061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.395032    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca" pod "ff9e0f75-9fd0-40a0-8875-d79524eb395f" (UID: "ff9e0f75-9fd0-40a0-8875-d79524eb395f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:23.395336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.395316    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:23.396461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.396433    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca" pod "ff9e0f75-9fd0-40a0-8875-d79524eb395f" (UID: "ff9e0f75-9fd0-40a0-8875-d79524eb395f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:23.396556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.396525    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c" pod "ff9e0f75-9fd0-40a0-8875-d79524eb395f" (UID: "ff9e0f75-9fd0-40a0-8875-d79524eb395f"). InnerVolumeSpecName "kube-api-access-fmn5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:23.496381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.496344    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmn5c\" (UniqueName: \"kubernetes.io/projected/ff9e0f75-9fd0-40a0-8875-d79524eb395f-kube-api-access-fmn5c\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:23.496381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:23.496378    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9e0f75-9fd0-40a0-8875-d79524eb395f-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:24.115833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:24.115807    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"
Oct 30 22:08:24.115985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:24.115806    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg" event={"ID":"ff9e0f75-9fd0-40a0-8875-d79524eb395f","Type":"ContainerDied","Data":"9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86"}
Oct 30 22:08:24.115985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:24.115927    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9057991eda2e0f6ffe2cd71a99efaf779a0cb783ef1d2a77ca8f29436b275b86"
Oct 30 22:08:26.121630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.121591    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:26.299531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.299493    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7376-8a35a5"]
Oct 30 22:08:26.300290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.300265    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ff9e0f75-9fd0-40a0-8875-d79524eb395f" containerName="revoke-keys"
Oct 30 22:08:26.300290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.300288    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9e0f75-9fd0-40a0-8875-d79524eb395f" containerName="revoke-keys"
Oct 30 22:08:26.300489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.300454    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9e0f75-9fd0-40a0-8875-d79524eb395f" containerName="revoke-keys"
Oct 30 22:08:26.304258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.304239    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:26.352615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.352584    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmr5\" (UniqueName: \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\") pod \"test-curl-7376-8a35a5\" (UID: \"1545edaf-ed45-4b00-951e-b94077f4a849\") " pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:26.364870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.364831    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-8a35a5"]
Oct 30 22:08:26.380387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.380298    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:26.406540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.406444    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbdpx\""
Oct 30 22:08:26.454325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.454290    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmr5\" (UniqueName: \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\") pod \"test-curl-7376-8a35a5\" (UID: \"1545edaf-ed45-4b00-951e-b94077f4a849\") " pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:26.512529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.512504    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmr5\" (UniqueName: \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\") pod \"test-curl-7376-8a35a5\" (UID: \"1545edaf-ed45-4b00-951e-b94077f4a849\") " pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:26.564852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.564814    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:26.624215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:26.624189    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:27.014592 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:08:27.014562    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1545edaf_ed45_4b00_951e_b94077f4a849.slice/crio-e9c0d7d942003739d28d11f5941e10c1d4444a921a836332a3fbbea46c62b00b WatchSource:0}: Error finding container e9c0d7d942003739d28d11f5941e10c1d4444a921a836332a3fbbea46c62b00b: Status 404 returned error can't find the container with id e9c0d7d942003739d28d11f5941e10c1d4444a921a836332a3fbbea46c62b00b
Oct 30 22:08:27.023765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.023737    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-8a35a5"]
Oct 30 22:08:27.063418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.063391    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"]
Oct 30 22:08:27.069167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.069145    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626739cf-bfgpr"]
Oct 30 22:08:27.160465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.160424    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-8a35a5" event={"ID":"1545edaf-ed45-4b00-951e-b94077f4a849","Type":"ContainerStarted","Data":"e9c0d7d942003739d28d11f5941e10c1d4444a921a836332a3fbbea46c62b00b"}
Oct 30 22:08:27.388188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.388113    4410 scope.go:122] "RemoveContainer" containerID="2308d3a5e7d09b082458f7a80bf0fcb21d48a4620628d8b137a03ce887ada4a4"
Oct 30 22:08:27.962198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:27.962170    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:28.172470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.172429    4410 generic.go:356] "Generic (PLEG): container finished" podID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerID="190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09" exitCode=0
Oct 30 22:08:28.173001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.172554    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerDied","Data":"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"}
Oct 30 22:08:28.173142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.173117    4410 scope.go:122] "RemoveContainer" containerID="190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"
Oct 30 22:08:28.174881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.174850    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:28.174994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.174928    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:28.174994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.174958    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:28.276208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:28.276179    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:29.020198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:29.020164    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145f98ec-bd2a-48bf-8275-05f1a33bb4d0" path="/var/lib/kubelet/pods/145f98ec-bd2a-48bf-8275-05f1a33bb4d0/volumes"
Oct 30 22:08:29.191971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:29.191923    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerStarted","Data":"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"}
Oct 30 22:08:29.475991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:29.475950    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:08:29.477014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:29.476987    4410 patch_prober.go:29] interesting pod/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:08:29.477102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:29.477058    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:08:30.472748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:30.472708    4410 patch_prober.go:29] interesting pod/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:08:30.473167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:30.472802    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:08:31.384675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:31.384095    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:31.475800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:31.473658    4410 patch_prober.go:29] interesting pod/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:08:31.475800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:31.473722    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:08:31.701645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:31.701608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:32.072258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:32.072165    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"]
Oct 30 22:08:32.119526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:32.119496    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa030e17-2hh2w"]
Oct 30 22:08:32.208223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:32.208190    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:08:32.472267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:32.472231    4410 patch_prober.go:29] interesting pod/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:08:32.472439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:32.472295    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:08:33.020230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:33.020197    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213f219c-be97-4ec7-8edb-db3a9a4f3bc0" path="/var/lib/kubelet/pods/213f219c-be97-4ec7-8edb-db3a9a4f3bc0/volumes"
Oct 30 22:08:33.472395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:33.472358    4410 patch_prober.go:29] interesting pod/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:08:33.472603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:33.472429    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:08:34.476872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:34.476838    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:08:34.477958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:34.477940    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:08:36.336094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:36.336056    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:36.574981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:36.574944    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:36.776722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:36.776688    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:37.084317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:37.084235    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"]
Oct 30 22:08:37.133407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:37.133382    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbc9857-wxb79"]
Oct 30 22:08:38.261332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:38.261277    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-8a35a5" event={"ID":"1545edaf-ed45-4b00-951e-b94077f4a849","Type":"ContainerStarted","Data":"e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0"}
Oct 30 22:08:38.380433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:38.380379    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7376-8a35a5" podStartSLOduration=1.7395874820000001 podStartE2EDuration="12.380364982s" totalImagesPullingTime="10.6407775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:38.379357177 +0000 UTC m=+1571.468094981" watchObservedRunningTime="2026-10-30 22:08:38.380364982 +0000 UTC m=+1571.469102785"
Oct 30 22:08:39.021921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:39.021889    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f6ce33-cd76-420c-b4ae-75501f0c46a0" path="/var/lib/kubelet/pods/f0f6ce33-cd76-420c-b4ae-75501f0c46a0/volumes"
Oct 30 22:08:40.886862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:40.886767    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:41.052526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.052493    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"]
Oct 30 22:08:41.102361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.102333    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"]
Oct 30 22:08:41.105848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.105826    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83534b8-6f8tl"]
Oct 30 22:08:41.109024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.108998    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370051c0-sgpn9"]
Oct 30 22:08:41.445814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.434200    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:08:41.445814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.441880    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.534145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.534114    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-97ca39-openshift-default-dockercfg-qfmxj\""
Oct 30 22:08:41.537080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.537045    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.537354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.537292    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.537581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.537555    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.537727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.537711    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.537980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.537948    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.538100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.538056    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:08:41.538235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.538155    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr79\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.538418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.538403    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.538540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.538524    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.538649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.538635    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.586805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.583581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:41.643732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.643694    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.643968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.643749    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.643968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.643823    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.643968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.643883    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.643968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.643920    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644012    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644042    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644058    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644083    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr79\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644154    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644424    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.644756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.644730    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.645121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.645097    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.645271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.645195    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.647939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.647895    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.660715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.660414    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.702526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.702451    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.703041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.703021    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr79\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\") pod \"e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") " pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.768034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.767993    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:41.813952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:41.813917    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:42.023724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.023604    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:42.220626 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:08:42.220585    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790f31ca_211c_4e39_9ab3_6158bbe697b6.slice/crio-ef7e7d45c1db0c6000604ec8b3c5101c9dd01cd6036d4572f0801d999e02f096 WatchSource:0}: Error finding container ef7e7d45c1db0c6000604ec8b3c5101c9dd01cd6036d4572f0801d999e02f096: Status 404 returned error can't find the container with id ef7e7d45c1db0c6000604ec8b3c5101c9dd01cd6036d4572f0801d999e02f096
Oct 30 22:08:42.221923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.221893    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:08:42.224025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.223656    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:42.224025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.223787    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:42.224025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.223866    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:42.224025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.223904    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:42.299996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.299963    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerStarted","Data":"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"}
Oct 30 22:08:42.300099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.300023    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerStarted","Data":"ef7e7d45c1db0c6000604ec8b3c5101c9dd01cd6036d4572f0801d999e02f096"}
Oct 30 22:08:42.437092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.437027    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podStartSLOduration=1.437005767 podStartE2EDuration="1.437005767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:42.431486984 +0000 UTC m=+1575.520224789" watchObservedRunningTime="2026-10-30 22:08:42.437005767 +0000 UTC m=+1575.525743572"
Oct 30 22:08:42.768607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:42.768572    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:43.020382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:43.020298    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c0374b-c9ef-4064-857f-130b03a5517e" path="/var/lib/kubelet/pods/01c0374b-c9ef-4064-857f-130b03a5517e/volumes"
Oct 30 22:08:43.020756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:43.020739    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880389eb-3712-40d6-9d86-7aa4fed801aa" path="/var/lib/kubelet/pods/880389eb-3712-40d6-9d86-7aa4fed801aa/volumes"
Oct 30 22:08:43.774244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:43.774216    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:44.315465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:44.315436    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:44.316278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:44.316257    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:08:45.440522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:45.440484    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:45.821654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:45.821575    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:47.456151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:47.456117    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:49.752678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:49.752642    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:49.880798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:49.879828    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7376-8a35a5"]
Oct 30 22:08:49.880798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:49.880048    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7376-8a35a5" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" containerName="test-curl-7376-8a35a5" containerID="cri-o://e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0" gracePeriod=2
Oct 30 22:08:49.884559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:49.884526    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7376-8a35a5"]
Oct 30 22:08:49.931138 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:49.931095    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-8a35a5\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:50.024572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.024453    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:50.259105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.259066    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:08:50.260016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.259994    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" containerName="test-curl-7376-8a35a5"
Oct 30 22:08:50.260118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.260017    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" containerName="test-curl-7376-8a35a5"
Oct 30 22:08:50.260172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.260144    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" containerName="test-curl-7376-8a35a5"
Oct 30 22:08:50.264061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.264041    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.324486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.324415    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:08:50.368909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.368858    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97ca39\""
Oct 30 22:08:50.369107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.369079    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97ca39-dockercfg-6vq7r\""
Oct 30 22:08:50.394745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.394705    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2zh\" (UniqueName: \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.395068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.395047    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.444148 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:50.444109    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-8a35a5\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:50.497005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.496966    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2zh\" (UniqueName: \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.497210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.497020    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.498045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.498009    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.536181 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:50.536140    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:08:50.557212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.557174    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2zh\" (UniqueName: \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\") pod \"payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.578912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.578844    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:50.842133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.842062    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"]
Oct 30 22:08:50.847865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.847834    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:50.898883 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:50.898828    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-8a35a5\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:50.900718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.900696    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:50.900900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.900878    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:50.900982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.900951    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:50.901050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.900968    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:08:50.904263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:50.904242    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"]
Oct 30 22:08:51.004765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.004738    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58zm\" (UniqueName: \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.004941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.004908    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.005074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.005055    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.031680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.031645    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:08:51.036550 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:08:51.036524    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6a65bd_8e63_41fe_bebf_ade4cbc232ac.slice/crio-f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0 WatchSource:0}: Error finding container f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0: Status 404 returned error can't find the container with id f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0
Oct 30 22:08:51.106297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.106267    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58zm\" (UniqueName: \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.106407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.106363    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.106510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.106495    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.107167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.107144    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.108502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.108483    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.168588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.168567    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58zm\" (UniqueName: \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.370492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.370407    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" event={"ID":"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac","Type":"ContainerStarted","Data":"72636161062bb84c8c1c54b105e7b15c3bddbb552e049489672ffa8da884e9d7"}
Oct 30 22:08:51.370492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.370464    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" event={"ID":"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac","Type":"ContainerStarted","Data":"f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0"}
Oct 30 22:08:51.370705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.370594    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:08:51.462051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.462021    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:51.487433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.487378    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" podStartSLOduration=1.487359168 podStartE2EDuration="1.487359168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:51.486891798 +0000 UTC m=+1584.575629602" watchObservedRunningTime="2026-10-30 22:08:51.487359168 +0000 UTC m=+1584.576096972"
Oct 30 22:08:51.848934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:51.848903    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"]
Oct 30 22:08:52.126862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.126839    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:52.224618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.224594    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\" (UniqueName: \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\") pod \"1545edaf-ed45-4b00-951e-b94077f4a849\" (UID: \"1545edaf-ed45-4b00-951e-b94077f4a849\") "
Oct 30 22:08:52.226443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.226419    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5" pod "1545edaf-ed45-4b00-951e-b94077f4a849" (UID: "1545edaf-ed45-4b00-951e-b94077f4a849"). InnerVolumeSpecName "kube-api-access-hpmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:52.325861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.325838    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmr5\" (UniqueName: \"kubernetes.io/projected/1545edaf-ed45-4b00-951e-b94077f4a849-kube-api-access-hpmr5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:52.378654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.378586    4410 generic.go:356] "Generic (PLEG): container finished" podID="1545edaf-ed45-4b00-951e-b94077f4a849" containerID="e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0" exitCode=137
Oct 30 22:08:52.378654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.378629    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:52.378825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.378683    4410 scope.go:122] "RemoveContainer" containerID="e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0"
Oct 30 22:08:52.380524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.380498    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d" event={"ID":"f261740f-84c2-422e-9dff-4df6eeae648e","Type":"ContainerStarted","Data":"0f8dadafe5c4f3046273b8455fd74e1065dfc9df15a648716224449a20c050fe"}
Oct 30 22:08:52.380653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.380551    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d" event={"ID":"f261740f-84c2-422e-9dff-4df6eeae648e","Type":"ContainerStarted","Data":"c8335c2b5951c0c04bb00f4a929f4f6f50081dd149bcf3799f0f489dd30fc085"}
Oct 30 22:08:52.389148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.389132    4410 scope.go:122] "RemoveContainer" containerID="e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0"
Oct 30 22:08:52.389392 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:52.389376    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0\": container with ID starting with e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0 not found: ID does not exist" containerID="e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0"
Oct 30 22:08:52.389435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.389399    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0"} err="failed to get container status \"e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0\": rpc error: code = NotFound desc = could not find container \"e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0\": container with ID starting with e835d4eefd8bb958f62fd3a542aedcebdf6004229f4ab5ef882bede9b92dd6d0 not found: ID does not exist"
Oct 30 22:08:52.504790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.504734    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d" podStartSLOduration=2.50471978 podStartE2EDuration="2.50471978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:52.503872616 +0000 UTC m=+1585.592610422" watchObservedRunningTime="2026-10-30 22:08:52.50471978 +0000 UTC m=+1585.593457583"
Oct 30 22:08:52.554415 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:08:52.554389    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-8a35a5\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" pod="opendatahub/test-curl-7376-8a35a5"
Oct 30 22:08:52.841493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:52.841454    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:53.020403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:53.020326    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1545edaf-ed45-4b00-951e-b94077f4a849" path="/var/lib/kubelet/pods/1545edaf-ed45-4b00-951e-b94077f4a849/volumes"
Oct 30 22:08:53.389030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:53.388997    4410 generic.go:356] "Generic (PLEG): container finished" podID="f261740f-84c2-422e-9dff-4df6eeae648e" containerID="0f8dadafe5c4f3046273b8455fd74e1065dfc9df15a648716224449a20c050fe" exitCode=0
Oct 30 22:08:53.389195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:53.389074    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d" event={"ID":"f261740f-84c2-422e-9dff-4df6eeae648e","Type":"ContainerDied","Data":"0f8dadafe5c4f3046273b8455fd74e1065dfc9df15a648716224449a20c050fe"}
Oct 30 22:08:54.058758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.058683    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"]
Oct 30 22:08:54.066123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.066098    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a92e35-j8zwz"]
Oct 30 22:08:54.542909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.542884    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:54.655753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.655675    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\") pod \"f261740f-84c2-422e-9dff-4df6eeae648e\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") "
Oct 30 22:08:54.655753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.655733    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\" (UniqueName: \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\") pod \"f261740f-84c2-422e-9dff-4df6eeae648e\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") "
Oct 30 22:08:54.656016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.655845    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\") pod \"f261740f-84c2-422e-9dff-4df6eeae648e\" (UID: \"f261740f-84c2-422e-9dff-4df6eeae648e\") "
Oct 30 22:08:54.656137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.656110    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca" pod "f261740f-84c2-422e-9dff-4df6eeae648e" (UID: "f261740f-84c2-422e-9dff-4df6eeae648e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:54.656587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.656569    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.657767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.657746    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca" pod "f261740f-84c2-422e-9dff-4df6eeae648e" (UID: "f261740f-84c2-422e-9dff-4df6eeae648e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:54.657970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.657953    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm" pod "f261740f-84c2-422e-9dff-4df6eeae648e" (UID: "f261740f-84c2-422e-9dff-4df6eeae648e"). InnerVolumeSpecName "kube-api-access-t58zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:54.757670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.757627    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58zm\" (UniqueName: \"kubernetes.io/projected/f261740f-84c2-422e-9dff-4df6eeae648e-kube-api-access-t58zm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.757670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:54.757667    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f261740f-84c2-422e-9dff-4df6eeae648e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:55.020390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:55.020297    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd871345-7092-48ab-bded-236815de68ea" path="/var/lib/kubelet/pods/bd871345-7092-48ab-bded-236815de68ea/volumes"
Oct 30 22:08:55.409059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:55.409028    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"
Oct 30 22:08:55.409059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:55.409039    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d" event={"ID":"f261740f-84c2-422e-9dff-4df6eeae648e","Type":"ContainerDied","Data":"c8335c2b5951c0c04bb00f4a929f4f6f50081dd149bcf3799f0f489dd30fc085"}
Oct 30 22:08:55.409468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:55.409096    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8335c2b5951c0c04bb00f4a929f4f6f50081dd149bcf3799f0f489dd30fc085"
Oct 30 22:08:58.644765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:08:58.644729    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:02.383615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:02.383587    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:09:02.782039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:02.782007    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:02.941957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:02.941921    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:03.905625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:03.905583    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:05.057856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:05.057820    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"]
Oct 30 22:09:05.107117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:05.107089    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdba4ee8-grksg"]
Oct 30 22:09:07.020744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:07.020710    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7d5fa2-0182-4213-9d3e-a34ba1f907db" path="/var/lib/kubelet/pods/ac7d5fa2-0182-4213-9d3e-a34ba1f907db/volumes"
Oct 30 22:09:09.043258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:09.043211    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:09.053559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:09.053523    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"]
Oct 30 22:09:09.101919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:09.101884    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c511335c-wvncq"]
Oct 30 22:09:10.165685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:10.165652    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.020741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:11.020703    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10b7d7b-d47b-4f07-8441-d9fed355ac77" path="/var/lib/kubelet/pods/d10b7d7b-d47b-4f07-8441-d9fed355ac77/volumes"
Oct 30 22:09:11.036242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:11.034318    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.600141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:11.600100    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.826472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:11.826434    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:14.644005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:14.643972    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.458486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:16.458450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.670978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:16.670940    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:19.049607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:19.049575    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"]
Oct 30 22:09:19.099054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:19.099021    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c5ed83-mp5jc"]
Oct 30 22:09:19.924481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:19.924428    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.526821 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:09:20.526739    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:09:20.702379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:20.702338    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.923822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:20.923751    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:21.022989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.022955    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30c9830-8f97-4b57-ba85-f6a0267369e4" path="/var/lib/kubelet/pods/f30c9830-8f97-4b57-ba85-f6a0267369e4/volumes"
Oct 30 22:09:21.095984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.095945    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:09:21.096916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.096888    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f261740f-84c2-422e-9dff-4df6eeae648e" containerName="revoke-keys"
Oct 30 22:09:21.096916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.096911    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f261740f-84c2-422e-9dff-4df6eeae648e" containerName="revoke-keys"
Oct 30 22:09:21.097128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.097116    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f261740f-84c2-422e-9dff-4df6eeae648e" containerName="revoke-keys"
Oct 30 22:09:21.108244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.108219    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.164331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.164295    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:09:21.210641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.210431    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2bc5e3-dockercfg-8qr7s\""
Oct 30 22:09:21.210641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.210522    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2bc5e3\""
Oct 30 22:09:21.213076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.213042    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.213542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.213497    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmm62\" (UniqueName: \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.273381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.272636    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:09:21.278045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.278018    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.314948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.314908    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.315124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.315030    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.315124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.315075    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpm5\" (UniqueName: \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.315249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.315207    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmm62\" (UniqueName: \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.315607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.315582    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.323297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.323266    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:09:21.375482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.375452    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmm62\" (UniqueName: \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\") pod \"payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.416699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.416669    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.416888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.416726    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpm5\" (UniqueName: \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.417422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.417402    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.433115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.433086    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:21.474059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.473964    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpm5\" (UniqueName: \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\") pod \"payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") " pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.594912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.593818    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:21.825384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.825345    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:09:21.826006 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:09:21.825981    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82214732_f8ab_433e_a9d6_eb93dd3fd8e3.slice/crio-88ec795dc49ff6cab3b4baf56ed3d99f14b91e374a9e794a528c1e0f308b77c9 WatchSource:0}: Error finding container 88ec795dc49ff6cab3b4baf56ed3d99f14b91e374a9e794a528c1e0f308b77c9: Status 404 returned error can't find the container with id 88ec795dc49ff6cab3b4baf56ed3d99f14b91e374a9e794a528c1e0f308b77c9
Oct 30 22:09:21.999569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:21.999543    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:09:22.000587 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:09:22.000559    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777fdffa_a0d5_465d_b000_bbfe819a983c.slice/crio-aecccebfbb7b6ca867c213afc24e39825b611c12318f029cbf0a30f18ed0ab59 WatchSource:0}: Error finding container aecccebfbb7b6ca867c213afc24e39825b611c12318f029cbf0a30f18ed0ab59: Status 404 returned error can't find the container with id aecccebfbb7b6ca867c213afc24e39825b611c12318f029cbf0a30f18ed0ab59
Oct 30 22:09:22.627538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.627498    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" event={"ID":"777fdffa-a0d5-465d-b000-bbfe819a983c","Type":"ContainerStarted","Data":"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"}
Oct 30 22:09:22.628071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.627566    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" event={"ID":"777fdffa-a0d5-465d-b000-bbfe819a983c","Type":"ContainerStarted","Data":"aecccebfbb7b6ca867c213afc24e39825b611c12318f029cbf0a30f18ed0ab59"}
Oct 30 22:09:22.628071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.627599    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:22.629198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.629168    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" event={"ID":"82214732-f8ab-433e-a9d6-eb93dd3fd8e3","Type":"ContainerStarted","Data":"64581b9e2e4129bae5ae328133fd83d06213f4bd953091c40b0ede4aa6548b5a"}
Oct 30 22:09:22.629328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.629225    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" event={"ID":"82214732-f8ab-433e-a9d6-eb93dd3fd8e3","Type":"ContainerStarted","Data":"88ec795dc49ff6cab3b4baf56ed3d99f14b91e374a9e794a528c1e0f308b77c9"}
Oct 30 22:09:22.629328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.629256    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:22.745385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.745343    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" podStartSLOduration=1.745330104 podStartE2EDuration="1.745330104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:22.741155178 +0000 UTC m=+1615.829892986" watchObservedRunningTime="2026-10-30 22:09:22.745330104 +0000 UTC m=+1615.834067906"
Oct 30 22:09:22.751274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:22.751245    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:25.277127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:25.277093    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:27.418801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.418740    4410 scope.go:122] "RemoveContainer" containerID="38784dfa9eba20e3f82ef1799369af8fc7a34651b6729ec8938fb1c1e958ee89"
Oct 30 22:09:27.431793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.431745    4410 scope.go:122] "RemoveContainer" containerID="808a6640c25bb0dd82834e9f14b989d2c3f1f178c07fce33dc1eaf1c1671df2a"
Oct 30 22:09:27.441326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.441302    4410 scope.go:122] "RemoveContainer" containerID="85e842658769d3f5f2b2ca5efc0931e997f7ade68e877b2b489ac768a1258729"
Oct 30 22:09:27.450201 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.450126    4410 scope.go:122] "RemoveContainer" containerID="e1edd68006874026be3be4cffb6d8d502a082e0d91cab78798ee40f66b207149"
Oct 30 22:09:27.458874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.458852    4410 scope.go:122] "RemoveContainer" containerID="8a7c18078b18e129557724a5e809438696760c43a53d081a53298260ea39feb3"
Oct 30 22:09:27.467168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:27.467143    4410 scope.go:122] "RemoveContainer" containerID="e5769e94d2f97933ea27d398158ff6dc06310427b8d5362c5d87caa81867ea16"
Oct 30 22:09:28.050117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:28.050063    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" podStartSLOduration=7.050050334 podStartE2EDuration="7.050050334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:22.860827167 +0000 UTC m=+1615.949564971" watchObservedRunningTime="2026-10-30 22:09:28.050050334 +0000 UTC m=+1621.138788136"
Oct 30 22:09:28.050622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:28.050605    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"]
Oct 30 22:09:28.099549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:28.099519    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3626c1fd-sp8cw"]
Oct 30 22:09:29.020870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:29.020836    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435d7350-72ab-4fc3-a7d5-58e1e464cbb9" path="/var/lib/kubelet/pods/435d7350-72ab-4fc3-a7d5-58e1e464cbb9/volumes"
Oct 30 22:09:30.192604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:30.192572    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:33.638939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:33.638905    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:09:33.639417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:33.638984    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:09:35.375670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:35.375630    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:37.689717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:37.689673    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.012553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.012519    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"]
Oct 30 22:09:40.018550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.018531    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.067354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.067327    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.067536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.067508    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbws\" (UniqueName: \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.067580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.067563    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.074582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.074556    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"]
Oct 30 22:09:40.119435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.119397    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:40.119435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.119424    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:40.119675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.119502    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:09:40.120790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.120760    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:40.169060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.169036    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.169180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.169148    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbws\" (UniqueName: \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.169180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.169174    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.169836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.169815    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.171142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.171123    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.229458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.229429    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbws\" (UniqueName: \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\") pod \"maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.333761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.333702    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:40.711505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.711478    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"]
Oct 30 22:09:40.713493 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:09:40.713460    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c06c2b8_cdb0_4e7f_8d03_746d0d21761d.slice/crio-b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6 WatchSource:0}: Error finding container b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6: Status 404 returned error can't find the container with id b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6
Oct 30 22:09:40.723957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.723930    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.771399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:40.771357    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7" event={"ID":"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d","Type":"ContainerStarted","Data":"b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6"}
Oct 30 22:09:41.781190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:41.781162    4410 generic.go:356] "Generic (PLEG): container finished" podID="4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" containerID="9b4ee2fc6f191b21dd45f51d6bdd9c4a519c21a49ceead8c98f659554f8e7d16" exitCode=0
Oct 30 22:09:41.781504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:41.781238    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7" event={"ID":"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d","Type":"ContainerDied","Data":"9b4ee2fc6f191b21dd45f51d6bdd9c4a519c21a49ceead8c98f659554f8e7d16"}
Oct 30 22:09:42.951798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:42.951764    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:43.005012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.004985    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\" (UniqueName: \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\") pod \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") "
Oct 30 22:09:43.005146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.005037    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\") pod \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") "
Oct 30 22:09:43.005224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.005161    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\") pod \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\" (UID: \"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d\") "
Oct 30 22:09:43.005511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.005486    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca" pod "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" (UID: "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:43.005854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.005833    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:43.007152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.007127    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca" pod "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" (UID: "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:43.007236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.007153    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws" pod "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" (UID: "4c06c2b8-cdb0-4e7f-8d03-746d0d21761d"). InnerVolumeSpecName "kube-api-access-fwbws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:43.066720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.066662    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"]
Oct 30 22:09:43.070501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.070479    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78f992f-5rsd4"]
Oct 30 22:09:43.107435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.107413    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbws\" (UniqueName: \"kubernetes.io/projected/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-kube-api-access-fwbws\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:43.107538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.107440    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:43.796426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.796389    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7" event={"ID":"4c06c2b8-cdb0-4e7f-8d03-746d0d21761d","Type":"ContainerDied","Data":"b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6"}
Oct 30 22:09:43.796426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.796408    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"
Oct 30 22:09:43.796637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:43.796439    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77c50724b372e27bd35a3dc47f063d68836febfa8502e695f8230e55495d9b6"
Oct 30 22:09:45.020701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:45.020666    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f04fba-ad84-4390-bec7-4a069953a8aa" path="/var/lib/kubelet/pods/d6f04fba-ad84-4390-bec7-4a069953a8aa/volumes"
Oct 30 22:09:46.278884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:46.278845    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:48.828402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:48.828342    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:50.522094 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:09:50.522057    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:09:51.543200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:51.543165    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:51.750153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:51.750116    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:51.915635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:51.915597    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:54.493334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:54.493300    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:54.653973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:54.653939    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:56.943576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:09:56.943544    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:00.499708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.499674    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"]
Oct 30 22:10:00.500297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.500283    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" containerName="revoke-keys"
Oct 30 22:10:00.500338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.500298    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" containerName="revoke-keys"
Oct 30 22:10:00.500392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.500383    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" containerName="revoke-keys"
Oct 30 22:10:00.504138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.504123    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.560825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.560766    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.560981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.560958    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.561078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.561061    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62fv\" (UniqueName: \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.605495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.605465    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:00.605495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.605482    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:00.605495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.605490    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:00.605761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.605582    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:10:00.609282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.609259    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"]
Oct 30 22:10:00.662303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.662276    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.662437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.662383    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62fv\" (UniqueName: \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.662543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.662527    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.663019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.662999    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.664860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.664835    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.720573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.720547    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62fv\" (UniqueName: \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\") pod \"maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:00.817519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:00.817451    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:01.194838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:01.194805    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"]
Oct 30 22:10:01.195659 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:10:01.195632    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14331eef_fb12_4bc8_81cd_83a783779c61.slice/crio-82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a WatchSource:0}: Error finding container 82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a: Status 404 returned error can't find the container with id 82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a
Oct 30 22:10:01.931741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:01.931702    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl" event={"ID":"14331eef-fb12-4bc8-81cd-83a783779c61","Type":"ContainerStarted","Data":"210861838a66c2edcb32376778ec9a1d8f46321fa765651ac3b8dc62f5181389"}
Oct 30 22:10:01.932150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:01.931765    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl" event={"ID":"14331eef-fb12-4bc8-81cd-83a783779c61","Type":"ContainerStarted","Data":"82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a"}
Oct 30 22:10:02.051513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:02.051465    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl" podStartSLOduration=2.051452478 podStartE2EDuration="2.051452478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:02.046653224 +0000 UTC m=+1655.135391029" watchObservedRunningTime="2026-10-30 22:10:02.051452478 +0000 UTC m=+1655.140190281"
Oct 30 22:10:02.072211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:02.072185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:02.939954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:02.939920    4410 generic.go:356] "Generic (PLEG): container finished" podID="14331eef-fb12-4bc8-81cd-83a783779c61" containerID="210861838a66c2edcb32376778ec9a1d8f46321fa765651ac3b8dc62f5181389" exitCode=0
Oct 30 22:10:02.940354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:02.939994    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl" event={"ID":"14331eef-fb12-4bc8-81cd-83a783779c61","Type":"ContainerDied","Data":"210861838a66c2edcb32376778ec9a1d8f46321fa765651ac3b8dc62f5181389"}
Oct 30 22:10:04.114338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.114315    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:04.213081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.213011    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\" (UniqueName: \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\") pod \"14331eef-fb12-4bc8-81cd-83a783779c61\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") "
Oct 30 22:10:04.213244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.213115    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\") pod \"14331eef-fb12-4bc8-81cd-83a783779c61\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") "
Oct 30 22:10:04.213244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.213189    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\") pod \"14331eef-fb12-4bc8-81cd-83a783779c61\" (UID: \"14331eef-fb12-4bc8-81cd-83a783779c61\") "
Oct 30 22:10:04.213438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.213415    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca" pod "14331eef-fb12-4bc8-81cd-83a783779c61" (UID: "14331eef-fb12-4bc8-81cd-83a783779c61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:04.213584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.213569    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:04.215144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.215117    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca" pod "14331eef-fb12-4bc8-81cd-83a783779c61" (UID: "14331eef-fb12-4bc8-81cd-83a783779c61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:04.215256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.215193    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv" pod "14331eef-fb12-4bc8-81cd-83a783779c61" (UID: "14331eef-fb12-4bc8-81cd-83a783779c61"). InnerVolumeSpecName "kube-api-access-s62fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:04.314694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.314663    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14331eef-fb12-4bc8-81cd-83a783779c61-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:04.314694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.314691    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62fv\" (UniqueName: \"kubernetes.io/projected/14331eef-fb12-4bc8-81cd-83a783779c61-kube-api-access-s62fv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:04.956196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.956165    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"
Oct 30 22:10:04.956452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.956170    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl" event={"ID":"14331eef-fb12-4bc8-81cd-83a783779c61","Type":"ContainerDied","Data":"82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a"}
Oct 30 22:10:04.956452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:04.956287    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82f9316af95e11334d6d1384ae816ae13eea1281da17ce275f59ded656d6513a"
Oct 30 22:10:05.076079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:05.076051    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:10:05.076266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:05.076240    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" containerID="cri-o://04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227" gracePeriod=30
Oct 30 22:10:06.601305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.601267    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"]
Oct 30 22:10:06.601899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.601882    4410 cpu_manager.go:414] "containerMap: removing container" podUID="14331eef-fb12-4bc8-81cd-83a783779c61" containerName="revoke-keys"
Oct 30 22:10:06.601899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.601899    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14331eef-fb12-4bc8-81cd-83a783779c61" containerName="revoke-keys"
Oct 30 22:10:06.602031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.602021    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="14331eef-fb12-4bc8-81cd-83a783779c61" containerName="revoke-keys"
Oct 30 22:10:06.605241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.605220    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.657957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.657930    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:06.658113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.658100    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:10:06.707656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.707631    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:06.707900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.707631    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:06.711718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.711689    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"]
Oct 30 22:10:06.746811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.746760    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.747046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.747027    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.747096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.747083    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxf6\" (UniqueName: \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.850125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.850071    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.850317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.850232    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxf6\" (UniqueName: \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.850377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.850361    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.851516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.851447    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.859212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.859182    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.911456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.911429    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxf6\" (UniqueName: \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\") pod \"maas-api-revoke-sub-5a77a61c-7gcvl\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:06.921271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:06.921249    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:07.301480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:07.301452    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"]
Oct 30 22:10:07.302448 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:10:07.302423    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa6b067_1c17_46a8_918f_ab71b781647a.slice/crio-e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35 WatchSource:0}: Error finding container e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35: Status 404 returned error can't find the container with id e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35
Oct 30 22:10:07.485910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:07.485875    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:07.980352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:07.980315    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl" event={"ID":"1fa6b067-1c17-46a8-918f-ab71b781647a","Type":"ContainerStarted","Data":"70a6e42c2349dd6a7f036cac1939136e01d825c84e971cafd46d677c862301b6"}
Oct 30 22:10:07.980799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:07.980380    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl" event={"ID":"1fa6b067-1c17-46a8-918f-ab71b781647a","Type":"ContainerStarted","Data":"e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35"}
Oct 30 22:10:08.097953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:08.097902    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl" podStartSLOduration=2.097888349 podStartE2EDuration="2.097888349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:08.096376947 +0000 UTC m=+1661.185114752" watchObservedRunningTime="2026-10-30 22:10:08.097888349 +0000 UTC m=+1661.186626181"
Oct 30 22:10:08.988760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:08.988726    4410 generic.go:356] "Generic (PLEG): container finished" podID="1fa6b067-1c17-46a8-918f-ab71b781647a" containerID="70a6e42c2349dd6a7f036cac1939136e01d825c84e971cafd46d677c862301b6" exitCode=0
Oct 30 22:10:08.989171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:08.988820    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl" event={"ID":"1fa6b067-1c17-46a8-918f-ab71b781647a","Type":"ContainerDied","Data":"70a6e42c2349dd6a7f036cac1939136e01d825c84e971cafd46d677c862301b6"}
Oct 30 22:10:10.210680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.210655    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:10.243513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.243481    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:10.243687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.243538    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:10.297205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.297175    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\" (UniqueName: \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\") pod \"1fa6b067-1c17-46a8-918f-ab71b781647a\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") "
Oct 30 22:10:10.297966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.297361    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\") pod \"1fa6b067-1c17-46a8-918f-ab71b781647a\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") "
Oct 30 22:10:10.297966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.297428    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\") pod \"1fa6b067-1c17-46a8-918f-ab71b781647a\" (UID: \"1fa6b067-1c17-46a8-918f-ab71b781647a\") "
Oct 30 22:10:10.298246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.298226    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca" pod "1fa6b067-1c17-46a8-918f-ab71b781647a" (UID: "1fa6b067-1c17-46a8-918f-ab71b781647a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.300352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.300321    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca" pod "1fa6b067-1c17-46a8-918f-ab71b781647a" (UID: "1fa6b067-1c17-46a8-918f-ab71b781647a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:10.300483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.300397    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6" pod "1fa6b067-1c17-46a8-918f-ab71b781647a" (UID: "1fa6b067-1c17-46a8-918f-ab71b781647a"). InnerVolumeSpecName "kube-api-access-2qxf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.318931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.318904    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:10:10.319105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.319085    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" containerID="cri-o://1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b" gracePeriod=30
Oct 30 22:10:10.331764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.331745    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:10:10.404227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.404192    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxf6\" (UniqueName: \"kubernetes.io/projected/1fa6b067-1c17-46a8-918f-ab71b781647a-kube-api-access-2qxf6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.404227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.404231    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.404474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.404247    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa6b067-1c17-46a8-918f-ab71b781647a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.505178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505149    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505240    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505326    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505365    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505396    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505435    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505502    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505549    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505602    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\") pod \"20125850-cbe5-4b46-b1b2-a41236d1eb55\" (UID: \"20125850-cbe5-4b46-b1b2-a41236d1eb55\") "
Oct 30 22:10:10.505753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505690    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.505753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505704    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.505920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.505825    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.506047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506024    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.506115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506042    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.506456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506434    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125850-cbe5-4b46-b1b2-a41236d1eb55-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.506527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506462    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.506527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506477    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.506527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506493    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.506527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.506507    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.507678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.507652    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "kube-api-access-vfm8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.507797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.507674    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:10.507797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.507681    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.507797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.507710    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token" pod "20125850-cbe5-4b46-b1b2-a41236d1eb55" (UID: "20125850-cbe5-4b46-b1b2-a41236d1eb55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.608222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.608149    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfm8m\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-kube-api-access-vfm8m\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.608222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.608178    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.608222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.608189    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.608222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:10.608199    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125850-cbe5-4b46-b1b2-a41236d1eb55-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:11.003059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.003024    4410 generic.go:356] "Generic (PLEG): container finished" podID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerID="04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227" exitCode=0
Oct 30 22:10:11.003227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.003087    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"
Oct 30 22:10:11.003227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.003109    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" event={"ID":"20125850-cbe5-4b46-b1b2-a41236d1eb55","Type":"ContainerDied","Data":"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"}
Oct 30 22:10:11.003227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.003186    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc" event={"ID":"20125850-cbe5-4b46-b1b2-a41236d1eb55","Type":"ContainerDied","Data":"1a6dda6ca4a2cf050f7855c5474c08679d672006dced17b4948d2ab566c2fd6e"}
Oct 30 22:10:11.003227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.003213    4410 scope.go:122] "RemoveContainer" containerID="04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"
Oct 30 22:10:11.004860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.004832    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl" event={"ID":"1fa6b067-1c17-46a8-918f-ab71b781647a","Type":"ContainerDied","Data":"e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35"}
Oct 30 22:10:11.004860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.004873    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e229c304e2a55bcdfd7fd7c9d730528483af71fedd0f5f3d14eec5204cca5c35"
Oct 30 22:10:11.005065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.004886    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"
Oct 30 22:10:11.014934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.014910    4410 scope.go:122] "RemoveContainer" containerID="04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"
Oct 30 22:10:11.015263 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:10:11.015239    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227\": container with ID starting with 04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227 not found: ID does not exist" containerID="04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"
Oct 30 22:10:11.015497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.015472    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227"} err="failed to get container status \"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227\": rpc error: code = NotFound desc = could not find container \"04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227\": container with ID starting with 04e79d631b35f2522dcf53f46ca4d149dc35ebd649ee74e5b53996c691052227 not found: ID does not exist"
Oct 30 22:10:11.167579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.167545    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:10:11.200955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:11.200923    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cgbtc"]
Oct 30 22:10:12.152980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:12.152946    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:10:12.436079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:12.435991    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:12.669019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:12.668973    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:12.864376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:12.864271    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:13.021667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:13.021635    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" path="/var/lib/kubelet/pods/20125850-cbe5-4b46-b1b2-a41236d1eb55/volumes"
Oct 30 22:10:13.042459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:13.042420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:13.587144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:13.587109    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:15.049478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.049445    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"]
Oct 30 22:10:15.097428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.097380    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:15.097699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.097664    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:15.099807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.099767    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d7c2c75-vr5tj"]
Oct 30 22:10:15.612473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.612448    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:10:15.696911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.696879    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.696911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.696917    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.696949    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697051    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697080    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697111    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697167    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.697351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697220    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697258    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697293    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\") pod \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\" (UID: \"3ac26be9-11e5-4d80-a7bf-37e37a6034b9\") "
Oct 30 22:10:15.697566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697444    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.697566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697470    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.697730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697576    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.697730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697662    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.698018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.697996    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.698099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.698023    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.698099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.698038    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.698099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.698047    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.698099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.698056    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.699460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.699438    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:15.699583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.699467    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.699756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.699734    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.699874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.699843    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92" pod "3ac26be9-11e5-4d80-a7bf-37e37a6034b9" (UID: "3ac26be9-11e5-4d80-a7bf-37e37a6034b9"). InnerVolumeSpecName "kube-api-access-jjp92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.799671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.799633    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.799671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.799674    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjp92\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-kube-api-access-jjp92\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.799903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.799690    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.799903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:15.799705    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac26be9-11e5-4d80-a7bf-37e37a6034b9-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:16.094837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.094798    4410 generic.go:356] "Generic (PLEG): container finished" podID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerID="1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b" exitCode=0
Oct 30 22:10:16.095312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.094865    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" event={"ID":"3ac26be9-11e5-4d80-a7bf-37e37a6034b9","Type":"ContainerDied","Data":"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"}
Oct 30 22:10:16.095312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.094944    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp" event={"ID":"3ac26be9-11e5-4d80-a7bf-37e37a6034b9","Type":"ContainerDied","Data":"f2492c8aba6b3b03882837eb5fe5e27a4c64ee8ca8ca53ee4f9ed278d238f362"}
Oct 30 22:10:16.095312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.094975    4410 scope.go:122] "RemoveContainer" containerID="1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"
Oct 30 22:10:16.095312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.095183    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"
Oct 30 22:10:16.110237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.109937    4410 scope.go:122] "RemoveContainer" containerID="1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"
Oct 30 22:10:16.110766 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:10:16.110686    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b\": container with ID starting with 1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b not found: ID does not exist" containerID="1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"
Oct 30 22:10:16.110766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.110728    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b"} err="failed to get container status \"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b\": rpc error: code = NotFound desc = could not find container \"1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b\": container with ID starting with 1d48ff942fa5f1893b58d526a88c81209e8db7b435c1326da66653188da79d5b not found: ID does not exist"
Oct 30 22:10:16.300552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.300520    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:10:16.306274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:16.306243    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-2j8mp"]
Oct 30 22:10:17.020727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.020694    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" path="/var/lib/kubelet/pods/3ac26be9-11e5-4d80-a7bf-37e37a6034b9/volumes"
Oct 30 22:10:17.021294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.021276    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccbab94-cd25-4042-a961-081d13529e9f" path="/var/lib/kubelet/pods/4ccbab94-cd25-4042-a961-081d13529e9f/volumes"
Oct 30 22:10:17.166869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.166828    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" podUID="cb621b3b-e125-4dbd-bd20-6331d7599779" containerName="payload-pre-processing" containerID="cri-o://078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36" gracePeriod=25
Oct 30 22:10:17.523276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.523251    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:10:17.630351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.630262    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\" (UniqueName: \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\") pod \"cb621b3b-e125-4dbd-bd20-6331d7599779\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") "
Oct 30 22:10:17.630528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.630510    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\") pod \"cb621b3b-e125-4dbd-bd20-6331d7599779\" (UID: \"cb621b3b-e125-4dbd-bd20-6331d7599779\") "
Oct 30 22:10:17.630897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.630875    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume" pod "cb621b3b-e125-4dbd-bd20-6331d7599779" (UID: "cb621b3b-e125-4dbd-bd20-6331d7599779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.631019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.631005    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb621b3b-e125-4dbd-bd20-6331d7599779-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.632491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.632470    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj" pod "cb621b3b-e125-4dbd-bd20-6331d7599779" (UID: "cb621b3b-e125-4dbd-bd20-6331d7599779"). InnerVolumeSpecName "kube-api-access-zrtkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.732435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:17.732392    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtkj\" (UniqueName: \"kubernetes.io/projected/cb621b3b-e125-4dbd-bd20-6331d7599779-kube-api-access-zrtkj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.111705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.111671    4410 generic.go:356] "Generic (PLEG): container finished" podID="cb621b3b-e125-4dbd-bd20-6331d7599779" containerID="078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36" exitCode=0
Oct 30 22:10:18.111869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.111729    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"
Oct 30 22:10:18.111869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.111758    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" event={"ID":"cb621b3b-e125-4dbd-bd20-6331d7599779","Type":"ContainerDied","Data":"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"}
Oct 30 22:10:18.111869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.111839    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45" event={"ID":"cb621b3b-e125-4dbd-bd20-6331d7599779","Type":"ContainerDied","Data":"776904a49d284c6065be14d089bce1307adc621dedea01214c524c633cee69cd"}
Oct 30 22:10:18.111869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.111862    4410 scope.go:122] "RemoveContainer" containerID="078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"
Oct 30 22:10:18.123119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.123100    4410 scope.go:122] "RemoveContainer" containerID="078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"
Oct 30 22:10:18.123384 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:10:18.123365    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36\": container with ID starting with 078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36 not found: ID does not exist" containerID="078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"
Oct 30 22:10:18.123436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.123393    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36"} err="failed to get container status \"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36\": rpc error: code = NotFound desc = could not find container \"078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36\": container with ID starting with 078ed9693b0e28d8cf85712e7d7acf3ebe3a0845b352e82062dfb460b63a6f36 not found: ID does not exist"
Oct 30 22:10:18.279756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.279723    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:10:18.285223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.285198    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-da0f3d-5975685759-qxn45"]
Oct 30 22:10:18.487622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:18.487591    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:19.020580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:19.020543    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb621b3b-e125-4dbd-bd20-6331d7599779" path="/var/lib/kubelet/pods/cb621b3b-e125-4dbd-bd20-6331d7599779/volumes"
Oct 30 22:10:20.523288 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:10:20.523250    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:10:23.095491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:23.095450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:23.896048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:23.896016    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:24.157171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.157143    4410 generic.go:356] "Generic (PLEG): container finished" podID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerID="f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617" exitCode=0
Oct 30 22:10:24.157600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.157217    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerDied","Data":"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"}
Oct 30 22:10:24.157648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.157615    4410 scope.go:122] "RemoveContainer" containerID="f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"
Oct 30 22:10:24.159030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.159002    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:24.159094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.159070    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:24.159132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:24.159101    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:25.177419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:25.177385    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerStarted","Data":"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"}
Oct 30 22:10:25.770299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:25.770091    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:10:25.771012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:25.770983    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:10:25.771235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:25.771201    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:10:26.770155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:26.770121    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:10:26.770574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:26.770174    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:10:27.520849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.520817    4410 scope.go:122] "RemoveContainer" containerID="5d977a55e963786710fa61529d4fa0727e228ff184eb7c37b16087267513c902"
Oct 30 22:10:27.530872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.530848    4410 scope.go:122] "RemoveContainer" containerID="fb59e219d13d042f2676425c9ffbf7fe54b6e2f6b769e7c3162b230e817baa2c"
Oct 30 22:10:27.539694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.539676    4410 scope.go:122] "RemoveContainer" containerID="e8018a9bab3dffae9dceb5ada16a76f520138efb3ec067ea5ec30f9ed36be976"
Oct 30 22:10:27.572279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.572250    4410 scope.go:122] "RemoveContainer" containerID="6b51ffbe718d2e50014a3f4466a897ec12756d4375272c945c7c160dde86593e"
Oct 30 22:10:27.580697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.580671    4410 scope.go:122] "RemoveContainer" containerID="7e8b82d2df7a73545f3fca00b0193d954d2fc1e68749ea641fe82a20772ddb54"
Oct 30 22:10:27.589683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.589665    4410 scope.go:122] "RemoveContainer" containerID="7bb79bc0eb3d08195e652863dc342a432fbf8b95026278397cef915f9b3ac55c"
Oct 30 22:10:27.599362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.599344    4410 scope.go:122] "RemoveContainer" containerID="6ca99f07a3eeead1910a1c50ee2b9f24aea14644ae3bc36d7ea234f9e072750a"
Oct 30 22:10:27.609181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.609166    4410 scope.go:122] "RemoveContainer" containerID="a452d293261df98a667ba8d68bc801dcb7e5df4bf78349236056ebbb33969342"
Oct 30 22:10:27.617991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.617971    4410 scope.go:122] "RemoveContainer" containerID="8e93cfe997b8eceb0b90454e001370c8ac16814b30a197b191de4116d6a74d27"
Oct 30 22:10:27.770185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.770156    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:10:27.770653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.770217    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:10:27.974074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.974043    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:10:27.974295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:27.974258    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy" containerID="cri-o://7d2b1f3017518e83ebd57279320e9701d40466d74779f69b21aca19a0a441166" gracePeriod=30
Oct 30 22:10:28.756914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:28.756877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:28.771199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:28.771157    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:10:28.771711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:28.771234    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:10:29.316271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:29.316236    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:10:29.772595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:29.772143    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:10:29.772595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:29.772208    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:10:30.772858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:30.772828    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:10:30.773896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:30.773874    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:10:31.048254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:31.048164    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"]
Oct 30 22:10:31.051492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:31.051470    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b53197-rr7xh"]
Oct 30 22:10:33.020341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.020309    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0154b759-6057-475e-8eff-644e71027e8e" path="/var/lib/kubelet/pods/0154b759-6057-475e-8eff-644e71027e8e/volumes"
Oct 30 22:10:33.232468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.232437    4410 generic.go:356] "Generic (PLEG): container finished" podID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerID="7d2b1f3017518e83ebd57279320e9701d40466d74779f69b21aca19a0a441166" exitCode=0
Oct 30 22:10:33.232586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.232514    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" event={"ID":"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9","Type":"ContainerDied","Data":"7d2b1f3017518e83ebd57279320e9701d40466d74779f69b21aca19a0a441166"}
Oct 30 22:10:33.232586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.232576    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2" event={"ID":"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9","Type":"ContainerDied","Data":"2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0"}
Oct 30 22:10:33.232661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.232600    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b287c5e41ca42ef35e174a845040360437fcc780d3d6e6eea263c461ec0ecd0"
Oct 30 22:10:33.238547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.238497    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:10:33.281020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.280997    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.281242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.281222    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:33.281412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.281400    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.382404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382381    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382465    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382533    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382580    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382611    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382646    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382672    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382657    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:33.382758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382699    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\") pod \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\" (UID: \"12cb44b3-e782-4c60-8323-4a7ee0cc7cb9\") "
Oct 30 22:10:33.382758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382729    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:33.383153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.382882    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:33.383153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.383081    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:33.383438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.383414    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.383596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.383568    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.383596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.383589    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.383792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.383604    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.384921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.384896    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:33.385014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.384978    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "kube-api-access-wvcwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:33.385014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.385006    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:33.385144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.385130    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy" pod "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" (UID: "12cb44b3-e782-4c60-8323-4a7ee0cc7cb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:33.485204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.485173    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcwp\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-kube-api-access-wvcwp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.485204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.485203    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.485385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.485216    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.485385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.485229    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.534475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:33.534399    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:34.239841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:34.239807    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"
Oct 30 22:10:34.410913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:34.410884    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:10:34.414216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:34.414190    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-xxzz2"]
Oct 30 22:10:35.019959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:35.019926    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" path="/var/lib/kubelet/pods/12cb44b3-e782-4c60-8323-4a7ee0cc7cb9/volumes"
Oct 30 22:10:38.485052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:38.485014    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:43.280052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:43.280018    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:48.275481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:48.275443    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:50.522710 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:10:50.522670    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:10:53.249741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:53.249710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.163852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:58.163816    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:10:58.432410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:58.432323    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.657081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:58.657029    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.820370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:58.820285    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:59.690525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:59.690490    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 22:10:59.691058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:10:59.690748    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" containerID="cri-o://55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1" gracePeriod=30
Oct 30 22:11:02.916523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:02.916474    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:03.164717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.164647    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" podUID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" containerName="payload-processing" containerID="cri-o://72636161062bb84c8c1c54b105e7b15c3bddbb552e049489672ffa8da884e9d7" gracePeriod=25
Oct 30 22:11:03.442022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.441946    4410 generic.go:356] "Generic (PLEG): container finished" podID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" containerID="72636161062bb84c8c1c54b105e7b15c3bddbb552e049489672ffa8da884e9d7" exitCode=0
Oct 30 22:11:03.442199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.442090    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" event={"ID":"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac","Type":"ContainerDied","Data":"72636161062bb84c8c1c54b105e7b15c3bddbb552e049489672ffa8da884e9d7"}
Oct 30 22:11:03.442199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.442138    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz" event={"ID":"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac","Type":"ContainerDied","Data":"f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0"}
Oct 30 22:11:03.442199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.442164    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ee0128b0fc95c8065ba37f5a077f0d094efa8d2d8cffadb3824b3d223354c0"
Oct 30 22:11:03.444900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.444872    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:11:03.596177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.596091    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\" (UniqueName: \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\") pod \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") "
Oct 30 22:11:03.596352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.596251    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\") pod \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\" (UID: \"ab6a65bd-8e63-41fe-bebf-ade4cbc232ac\") "
Oct 30 22:11:03.596583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.596558    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume" pod "ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" (UID: "ab6a65bd-8e63-41fe-bebf-ade4cbc232ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.596916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.596897    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.598861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.598823    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh" pod "ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" (UID: "ab6a65bd-8e63-41fe-bebf-ade4cbc232ac"). InnerVolumeSpecName "kube-api-access-6m2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.698373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:03.698344    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2zh\" (UniqueName: \"kubernetes.io/projected/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac-kube-api-access-6m2zh\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:04.449163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.449133    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"
Oct 30 22:11:04.510864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.510821    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.613087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.613051    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:11:04.662566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.662527    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-864dc78c4f-6l2dz"]
Oct 30 22:11:04.728686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.728656    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.847816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.846333    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bm2hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:11:04.847816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.846415    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:11:04.912017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.911977    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.985524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:04.985500    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 22:11:05.019489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.019460    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" path="/var/lib/kubelet/pods/ab6a65bd-8e63-41fe-bebf-ade4cbc232ac/volumes"
Oct 30 22:11:05.033860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.033834    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.033987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.033896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.033997    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034059    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034122    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034150    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034181    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034227    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034284    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034290    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\") pod \"8f313244-2f2a-40e1-8775-8146400ed562\" (UID: \"8f313244-2f2a-40e1-8775-8146400ed562\") "
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034429    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:05.034612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034510    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.035169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034722    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.035169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.034748    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.035292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.035264    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f313244-2f2a-40e1-8775-8146400ed562-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.035354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.035293    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.035354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.035307    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.035354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.035322    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.035354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.035336    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.036359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.036334    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "kube-api-access-9zq85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.044086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.037486    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:05.047877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.047847    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.047971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.047894    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token" pod "8f313244-2f2a-40e1-8775-8146400ed562" (UID: "8f313244-2f2a-40e1-8775-8146400ed562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.137224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.137181    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zq85\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-kube-api-access-9zq85\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.137224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.137219    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f313244-2f2a-40e1-8775-8146400ed562-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.137474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.137235    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f313244-2f2a-40e1-8775-8146400ed562-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.137474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.137248    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f313244-2f2a-40e1-8775-8146400ed562-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.140835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.140804    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"]
Oct 30 22:11:05.146207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.146176    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-wtn7d"]
Oct 30 22:11:05.462135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.462095    4410 generic.go:356] "Generic (PLEG): container finished" podID="8f313244-2f2a-40e1-8775-8146400ed562" containerID="55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1" exitCode=0
Oct 30 22:11:05.462629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.462177    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" event={"ID":"8f313244-2f2a-40e1-8775-8146400ed562","Type":"ContainerDied","Data":"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"}
Oct 30 22:11:05.462629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.462184    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"
Oct 30 22:11:05.462629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.462305    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx" event={"ID":"8f313244-2f2a-40e1-8775-8146400ed562","Type":"ContainerDied","Data":"1dfad838a334559175b2dbf3f0faf51a97b08f909231c26a51589023566bc8ed"}
Oct 30 22:11:05.462629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.462343    4410 scope.go:122] "RemoveContainer" containerID="55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"
Oct 30 22:11:05.473415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.473397    4410 scope.go:122] "RemoveContainer" containerID="55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"
Oct 30 22:11:05.473704 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:11:05.473685    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1\": container with ID starting with 55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1 not found: ID does not exist" containerID="55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"
Oct 30 22:11:05.473751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.473713    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1"} err="failed to get container status \"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1\": rpc error: code = NotFound desc = could not find container \"55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1\": container with ID starting with 55c86aa6da468d6c734b5a047fca040f1584f9fe307adb7c30ed2654023eb4b1 not found: ID does not exist"
Oct 30 22:11:05.630591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.630558    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 22:11:05.635859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:05.635833    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bm2hx"]
Oct 30 22:11:07.020299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:07.020265    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f313244-2f2a-40e1-8775-8146400ed562" path="/var/lib/kubelet/pods/8f313244-2f2a-40e1-8775-8146400ed562/volumes"
Oct 30 22:11:07.020744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:07.020729    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f261740f-84c2-422e-9dff-4df6eeae648e" path="/var/lib/kubelet/pods/f261740f-84c2-422e-9dff-4df6eeae648e/volumes"
Oct 30 22:11:09.376506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:09.376474    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:10.998640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:10.998598    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:11.305240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:11.305139    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:11.498700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:11.498659    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"]
Oct 30 22:11:11.512521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:11.512483    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:11.548313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:11.548285    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ac4a7-98wk7"]
Oct 30 22:11:13.019686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:13.019651    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c06c2b8-cdb0-4e7f-8d03-746d0d21761d" path="/var/lib/kubelet/pods/4c06c2b8-cdb0-4e7f-8d03-746d0d21761d/volumes"
Oct 30 22:11:16.693766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:16.693730    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:18.569928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:18.569894    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:20.524166 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:11:20.524126    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:11:21.911480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:21.911442    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:22.072257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:22.072225    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:23.461903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:23.461865    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:26.559471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:26.559417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:27.716125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:27.716095    4410 scope.go:122] "RemoveContainer" containerID="f7f9f7b4cc4d0ac1d598d17ae4c7154fd8436354eae6fe362e0c348fbea60269"
Oct 30 22:11:27.725234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:27.725212    4410 scope.go:122] "RemoveContainer" containerID="ebe529d1ff3e3dc60b11a7a3037309e37b02c7d42b86f8e71b8c8fd25807c7dc"
Oct 30 22:11:27.733520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:27.733499    4410 scope.go:122] "RemoveContainer" containerID="7d2b1f3017518e83ebd57279320e9701d40466d74779f69b21aca19a0a441166"
Oct 30 22:11:28.140211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:28.140176    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.385432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:28.385396    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.568967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:28.568877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.668808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:28.668760    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"]
Oct 30 22:11:28.718031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:28.717998    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-da0f3d-2h9dl"]
Oct 30 22:11:29.021767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:29.021731    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14331eef-fb12-4bc8-81cd-83a783779c61" path="/var/lib/kubelet/pods/14331eef-fb12-4bc8-81cd-83a783779c61/volumes"
Oct 30 22:11:29.074382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:29.074347    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:29.284477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:29.284390    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:29.479797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:29.479221    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:30.265650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.265619    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"]
Oct 30 22:11:30.266181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266167    4410 cpu_manager.go:414] "containerMap: removing container" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy"
Oct 30 22:11:30.266229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266182    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy"
Oct 30 22:11:30.266229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266198    4410 cpu_manager.go:414] "containerMap: removing container" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy"
Oct 30 22:11:30.266229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266203    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy"
Oct 30 22:11:30.266229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266214    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cb621b3b-e125-4dbd-bd20-6331d7599779" containerName="payload-pre-processing"
Oct 30 22:11:30.266229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266219    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb621b3b-e125-4dbd-bd20-6331d7599779" containerName="payload-pre-processing"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266237    4410 cpu_manager.go:414] "containerMap: removing container" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266243    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266255    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" containerName="payload-processing"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266260    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" containerName="payload-processing"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266268    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266274    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266293    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1fa6b067-1c17-46a8-918f-ab71b781647a" containerName="revoke-keys"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266298    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa6b067-1c17-46a8-918f-ab71b781647a" containerName="revoke-keys"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266373    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6a65bd-8e63-41fe-bebf-ade4cbc232ac" containerName="payload-processing"
Oct 30 22:11:30.266380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266382    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f313244-2f2a-40e1-8775-8146400ed562" containerName="istio-proxy"
Oct 30 22:11:30.266672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266389    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="20125850-cbe5-4b46-b1b2-a41236d1eb55" containerName="istio-proxy"
Oct 30 22:11:30.266672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266396    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac26be9-11e5-4d80-a7bf-37e37a6034b9" containerName="istio-proxy"
Oct 30 22:11:30.266672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266402    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb621b3b-e125-4dbd-bd20-6331d7599779" containerName="payload-pre-processing"
Oct 30 22:11:30.266672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266410    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa6b067-1c17-46a8-918f-ab71b781647a" containerName="revoke-keys"
Oct 30 22:11:30.266672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.266417    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cb44b3-e782-4c60-8323-4a7ee0cc7cb9" containerName="istio-proxy"
Oct 30 22:11:30.271509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.271493    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.290305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.290277    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.290417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.290386    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpfb\" (UniqueName: \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.290463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.290434    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.328525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.328502    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:30.329477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.329457    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:11:30.330289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.330231    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:30.333714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.333691    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"]
Oct 30 22:11:30.377966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.377947    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:30.391691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.391666    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpfb\" (UniqueName: \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.391803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.391716    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.391893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.391880    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.392493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.392473    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.394075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.394056    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.450370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.450341    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpfb\" (UniqueName: \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\") pod \"maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.584896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.584830    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:30.971220 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:11:30.971179    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f6abf8_a5ed_44b2_b34a_5377db608d81.slice/crio-91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a WatchSource:0}: Error finding container 91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a: Status 404 returned error can't find the container with id 91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a
Oct 30 22:11:30.971350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:30.971225    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"]
Oct 30 22:11:31.646729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:31.646695    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw" event={"ID":"75f6abf8-a5ed-44b2-b34a-5377db608d81","Type":"ContainerStarted","Data":"e306a8330431751bf71a346a6e7badcecde9e560ef54eb14e029d81f5028f15b"}
Oct 30 22:11:31.647221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:31.646756    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw" event={"ID":"75f6abf8-a5ed-44b2-b34a-5377db608d81","Type":"ContainerStarted","Data":"91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a"}
Oct 30 22:11:31.822053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:31.821992    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw" podStartSLOduration=1.8219784940000001 podStartE2EDuration="1.821978494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:31.772143828 +0000 UTC m=+1744.860881631" watchObservedRunningTime="2026-10-30 22:11:31.821978494 +0000 UTC m=+1744.910716298"
Oct 30 22:11:32.654030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:32.653997    4410 generic.go:356] "Generic (PLEG): container finished" podID="75f6abf8-a5ed-44b2-b34a-5377db608d81" containerID="e306a8330431751bf71a346a6e7badcecde9e560ef54eb14e029d81f5028f15b" exitCode=0
Oct 30 22:11:32.654451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:32.654061    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw" event={"ID":"75f6abf8-a5ed-44b2-b34a-5377db608d81","Type":"ContainerDied","Data":"e306a8330431751bf71a346a6e7badcecde9e560ef54eb14e029d81f5028f15b"}
Oct 30 22:11:33.798825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.798802    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:33.829173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.829148    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\") pod \"75f6abf8-a5ed-44b2-b34a-5377db608d81\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") "
Oct 30 22:11:33.829298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.829220    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\" (UniqueName: \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\") pod \"75f6abf8-a5ed-44b2-b34a-5377db608d81\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") "
Oct 30 22:11:33.829375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.829308    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\") pod \"75f6abf8-a5ed-44b2-b34a-5377db608d81\" (UID: \"75f6abf8-a5ed-44b2-b34a-5377db608d81\") "
Oct 30 22:11:33.829561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.829526    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca" pod "75f6abf8-a5ed-44b2-b34a-5377db608d81" (UID: "75f6abf8-a5ed-44b2-b34a-5377db608d81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:33.829886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.829868    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.831317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.831295    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb" pod "75f6abf8-a5ed-44b2-b34a-5377db608d81" (UID: "75f6abf8-a5ed-44b2-b34a-5377db608d81"). InnerVolumeSpecName "kube-api-access-bvpfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:33.831399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.831349    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca" pod "75f6abf8-a5ed-44b2-b34a-5377db608d81" (UID: "75f6abf8-a5ed-44b2-b34a-5377db608d81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:33.931277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.931203    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvpfb\" (UniqueName: \"kubernetes.io/projected/75f6abf8-a5ed-44b2-b34a-5377db608d81-kube-api-access-bvpfb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.931277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:33.931229    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6abf8-a5ed-44b2-b34a-5377db608d81-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.138524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:34.138462    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.667175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:34.667149    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"
Oct 30 22:11:34.667353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:34.667149    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw" event={"ID":"75f6abf8-a5ed-44b2-b34a-5377db608d81","Type":"ContainerDied","Data":"91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a"}
Oct 30 22:11:34.667353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:34.667276    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91101cc2151fc71c86cc1c6dbddbf0e5449f4c4d8c64a4e7323027f4aa274b2a"
Oct 30 22:11:36.109464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:36.109430    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.412759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:36.412722    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.870425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:36.870339    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:37.072282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.072250    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"]
Oct 30 22:11:37.121187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.121126    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba22b57-fsn9s"]
Oct 30 22:11:37.678735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.678702    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"]
Oct 30 22:11:37.679288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.679274    4410 cpu_manager.go:414] "containerMap: removing container" podUID="75f6abf8-a5ed-44b2-b34a-5377db608d81" containerName="revoke-keys"
Oct 30 22:11:37.679330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.679289    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f6abf8-a5ed-44b2-b34a-5377db608d81" containerName="revoke-keys"
Oct 30 22:11:37.679404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.679395    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f6abf8-a5ed-44b2-b34a-5377db608d81" containerName="revoke-keys"
Oct 30 22:11:37.682892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.682876    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.786669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.786598    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:37.786669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.786630    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:37.786669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.786641    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:11:37.786669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.786630    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:37.790392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.790363    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"]
Oct 30 22:11:37.799278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.799252    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.799420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.799362    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.799493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.799457    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzsl2\" (UniqueName: \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.900338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.900313    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.900480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.900379    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzsl2\" (UniqueName: \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.900480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.900463    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.901082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.901059    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.902665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.902643    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.957709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.957643    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzsl2\" (UniqueName: \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\") pod \"maas-api-revoke-keys-e2e-api-b-479363-csxj9\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:37.996034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:37.995996    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:38.412574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:38.412545    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"]
Oct 30 22:11:38.413177 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:11:38.413150    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2539564f_af38_40a1_8617_ca6aafa70e62.slice/crio-71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662 WatchSource:0}: Error finding container 71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662: Status 404 returned error can't find the container with id 71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662
Oct 30 22:11:38.741130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:38.741058    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9" event={"ID":"2539564f-af38-40a1-8617-ca6aafa70e62","Type":"ContainerStarted","Data":"5c1b8f4ac91f09646adc0b1efe6440073ebf7f4301e47a2de2066514bf3f04a6"}
Oct 30 22:11:38.741130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:38.741111    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9" event={"ID":"2539564f-af38-40a1-8617-ca6aafa70e62","Type":"ContainerStarted","Data":"71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662"}
Oct 30 22:11:38.867809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:38.867739    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9" podStartSLOduration=1.867725068 podStartE2EDuration="1.867725068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:38.86439468 +0000 UTC m=+1751.953132496" watchObservedRunningTime="2026-10-30 22:11:38.867725068 +0000 UTC m=+1751.956462870"
Oct 30 22:11:39.019510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:39.019440    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5324f5a4-40f5-4caf-85e4-2f2a12546b48" path="/var/lib/kubelet/pods/5324f5a4-40f5-4caf-85e4-2f2a12546b48/volumes"
Oct 30 22:11:39.183516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:39.183455    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.749331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:39.749295    4410 generic.go:356] "Generic (PLEG): container finished" podID="2539564f-af38-40a1-8617-ca6aafa70e62" containerID="5c1b8f4ac91f09646adc0b1efe6440073ebf7f4301e47a2de2066514bf3f04a6" exitCode=0
Oct 30 22:11:39.749745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:39.749394    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9" event={"ID":"2539564f-af38-40a1-8617-ca6aafa70e62","Type":"ContainerDied","Data":"5c1b8f4ac91f09646adc0b1efe6440073ebf7f4301e47a2de2066514bf3f04a6"}
Oct 30 22:11:40.915165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:40.915139    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:41.037411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.037341    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\" (UniqueName: \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\") pod \"2539564f-af38-40a1-8617-ca6aafa70e62\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") "
Oct 30 22:11:41.037411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.037382    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\") pod \"2539564f-af38-40a1-8617-ca6aafa70e62\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") "
Oct 30 22:11:41.037594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.037428    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\") pod \"2539564f-af38-40a1-8617-ca6aafa70e62\" (UID: \"2539564f-af38-40a1-8617-ca6aafa70e62\") "
Oct 30 22:11:41.038012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.037986    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca" pod "2539564f-af38-40a1-8617-ca6aafa70e62" (UID: "2539564f-af38-40a1-8617-ca6aafa70e62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:41.039618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.039590    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca" pod "2539564f-af38-40a1-8617-ca6aafa70e62" (UID: "2539564f-af38-40a1-8617-ca6aafa70e62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:41.039893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.039869    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2" pod "2539564f-af38-40a1-8617-ca6aafa70e62" (UID: "2539564f-af38-40a1-8617-ca6aafa70e62"). InnerVolumeSpecName "kube-api-access-tzsl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:41.140064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.140032    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzsl2\" (UniqueName: \"kubernetes.io/projected/2539564f-af38-40a1-8617-ca6aafa70e62-kube-api-access-tzsl2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.140196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.140068    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.140196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.140087    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2539564f-af38-40a1-8617-ca6aafa70e62-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.362500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.362417    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"]
Oct 30 22:11:41.363000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.362982    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2539564f-af38-40a1-8617-ca6aafa70e62" containerName="revoke-keys"
Oct 30 22:11:41.363059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.363001    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2539564f-af38-40a1-8617-ca6aafa70e62" containerName="revoke-keys"
Oct 30 22:11:41.363137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.363127    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2539564f-af38-40a1-8617-ca6aafa70e62" containerName="revoke-keys"
Oct 30 22:11:41.366613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.366592    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.425763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.425735    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"]
Oct 30 22:11:41.443338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.443310    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shd5l\" (UniqueName: \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.443471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.443377    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.443471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.443459    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.544405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.544374    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shd5l\" (UniqueName: \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.544560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.544434    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.544560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.544476    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.545132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.545109    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.546976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.546952    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.602437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.602406    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shd5l\" (UniqueName: \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\") pod \"maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.680557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.680530    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:41.766165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.766137    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"
Oct 30 22:11:41.766531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.766132    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9" event={"ID":"2539564f-af38-40a1-8617-ca6aafa70e62","Type":"ContainerDied","Data":"71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662"}
Oct 30 22:11:41.766531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:41.766502    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71694ee79bc3034b7e43b91407b95d6df47508ba977a522be859a3d8333ad662"
Oct 30 22:11:42.065818 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:11:42.057968    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df2c45c_bca2_4a0d_a598_96bac3ec7db0.slice/crio-c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b WatchSource:0}: Error finding container c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b: Status 404 returned error can't find the container with id c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b
Oct 30 22:11:42.066564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:42.065937    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"]
Oct 30 22:11:42.775668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:42.775628    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx" event={"ID":"2df2c45c-bca2-4a0d-a598-96bac3ec7db0","Type":"ContainerStarted","Data":"61eedaf90d5f4bdb551316efe8fe5a466f259f590dd221204bcaa9a2ebfbee4d"}
Oct 30 22:11:42.775868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:42.775685    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx" event={"ID":"2df2c45c-bca2-4a0d-a598-96bac3ec7db0","Type":"ContainerStarted","Data":"c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b"}
Oct 30 22:11:42.892372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:42.892324    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx" podStartSLOduration=1.892312128 podStartE2EDuration="1.892312128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:42.890224392 +0000 UTC m=+1755.978962207" watchObservedRunningTime="2026-10-30 22:11:42.892312128 +0000 UTC m=+1755.981049931"
Oct 30 22:11:43.241181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:43.241153    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:11:43.531440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:43.531333    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:43.783898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:43.783812    4410 generic.go:356] "Generic (PLEG): container finished" podID="2df2c45c-bca2-4a0d-a598-96bac3ec7db0" containerID="61eedaf90d5f4bdb551316efe8fe5a466f259f590dd221204bcaa9a2ebfbee4d" exitCode=0
Oct 30 22:11:43.784050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:43.783896    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx" event={"ID":"2df2c45c-bca2-4a0d-a598-96bac3ec7db0","Type":"ContainerDied","Data":"61eedaf90d5f4bdb551316efe8fe5a466f259f590dd221204bcaa9a2ebfbee4d"}
Oct 30 22:11:43.833736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:43.833705    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:44.622557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:44.622520    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:45.061052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.061023    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:45.146025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.145997    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\") pod \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") "
Oct 30 22:11:45.146165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.146128    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\") pod \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") "
Oct 30 22:11:45.146277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.146265    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\" (UniqueName: \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\") pod \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\" (UID: \"2df2c45c-bca2-4a0d-a598-96bac3ec7db0\") "
Oct 30 22:11:45.146364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.146339    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca" pod "2df2c45c-bca2-4a0d-a598-96bac3ec7db0" (UID: "2df2c45c-bca2-4a0d-a598-96bac3ec7db0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:45.146664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.146643    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.147967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.147949    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca" pod "2df2c45c-bca2-4a0d-a598-96bac3ec7db0" (UID: "2df2c45c-bca2-4a0d-a598-96bac3ec7db0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:45.148198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.148184    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l" pod "2df2c45c-bca2-4a0d-a598-96bac3ec7db0" (UID: "2df2c45c-bca2-4a0d-a598-96bac3ec7db0"). InnerVolumeSpecName "kube-api-access-shd5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:45.247632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.247555    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.247632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.247587    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shd5l\" (UniqueName: \"kubernetes.io/projected/2df2c45c-bca2-4a0d-a598-96bac3ec7db0-kube-api-access-shd5l\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.798162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.798081    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"
Oct 30 22:11:45.798162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.798112    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx" event={"ID":"2df2c45c-bca2-4a0d-a598-96bac3ec7db0","Type":"ContainerDied","Data":"c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b"}
Oct 30 22:11:45.798162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:45.798160    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6f6f3099a1d588092767cf0e2e465878c3914dd3a99b061c5f9666c00d7649b"
Oct 30 22:11:47.531911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:47.531877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:47.775094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:47.775054    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:48.241531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.241489    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" podUID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" containerName="payload-pre-processing" containerID="cri-o://b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d" gracePeriod=25
Oct 30 22:11:48.517630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.517604    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:11:48.590103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.590067    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\") pod \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") "
Oct 30 22:11:48.590494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.590279    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\" (UniqueName: \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\") pod \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\" (UID: \"64d7fee0-5f25-405a-aa09-1f7d569c78b8\") "
Oct 30 22:11:48.590494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.590371    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume" pod "64d7fee0-5f25-405a-aa09-1f7d569c78b8" (UID: "64d7fee0-5f25-405a-aa09-1f7d569c78b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:48.590755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.590732    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d7fee0-5f25-405a-aa09-1f7d569c78b8-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:48.592365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.592342    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw" pod "64d7fee0-5f25-405a-aa09-1f7d569c78b8" (UID: "64d7fee0-5f25-405a-aa09-1f7d569c78b8"). InnerVolumeSpecName "kube-api-access-j2rsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:48.692549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.692513    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2rsw\" (UniqueName: \"kubernetes.io/projected/64d7fee0-5f25-405a-aa09-1f7d569c78b8-kube-api-access-j2rsw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:48.827999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.827921    4410 generic.go:356] "Generic (PLEG): container finished" podID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" containerID="b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d" exitCode=0
Oct 30 22:11:48.827999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.827975    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"
Oct 30 22:11:48.828228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.827980    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" event={"ID":"64d7fee0-5f25-405a-aa09-1f7d569c78b8","Type":"ContainerDied","Data":"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"}
Oct 30 22:11:48.828228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.828129    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6" event={"ID":"64d7fee0-5f25-405a-aa09-1f7d569c78b8","Type":"ContainerDied","Data":"2fe39beedfbf51d9da7fdaeb973f278b92afca53fbffc31b7f9e4747a2982565"}
Oct 30 22:11:48.828228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.828160    4410 scope.go:122] "RemoveContainer" containerID="b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"
Oct 30 22:11:48.839212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.839188    4410 scope.go:122] "RemoveContainer" containerID="b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"
Oct 30 22:11:48.839503 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:11:48.839483    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d\": container with ID starting with b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d not found: ID does not exist" containerID="b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"
Oct 30 22:11:48.839562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.839514    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d"} err="failed to get container status \"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d\": rpc error: code = NotFound desc = could not find container \"b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d\": container with ID starting with b200e71d6a4d6ce1a0672715fb9fb3699fddc8e1cb677bce37f0ef36d2e1858d not found: ID does not exist"
Oct 30 22:11:48.996949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:48.996914    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:11:49.002591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:49.002562    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-02a8fa-54db49f4d7-qqrb6"]
Oct 30 22:11:49.020214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:49.020180    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" path="/var/lib/kubelet/pods/64d7fee0-5f25-405a-aa09-1f7d569c78b8/volumes"
Oct 30 22:11:50.138297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:50.138265    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:50.442718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:50.442683    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:50.524240 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:11:50.524210    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:11:50.797234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:50.797149    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.216960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:54.216924    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:55.452407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:55.452369    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:56.748176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:56.748141    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:11:57.056706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:57.056617    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:57.308099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:57.308006    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.082177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:11:58.082134    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:01.748746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:01.748688    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" podUID="5f1bf09f-42c7-4382-9eef-070316319ab5" containerName="payload-pre-processing" containerID="cri-o://25a0b165baa76471724d28dfa1b74457496ef1c502e75d59986d72565d3531d9" gracePeriod=25
Oct 30 22:12:01.937929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:01.937760    4410 generic.go:356] "Generic (PLEG): container finished" podID="5f1bf09f-42c7-4382-9eef-070316319ab5" containerID="25a0b165baa76471724d28dfa1b74457496ef1c502e75d59986d72565d3531d9" exitCode=0
Oct 30 22:12:01.937929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:01.937823    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" event={"ID":"5f1bf09f-42c7-4382-9eef-070316319ab5","Type":"ContainerDied","Data":"25a0b165baa76471724d28dfa1b74457496ef1c502e75d59986d72565d3531d9"}
Oct 30 22:12:02.018520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.018461    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:12:02.074900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.074872    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\" (UniqueName: \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\") pod \"5f1bf09f-42c7-4382-9eef-070316319ab5\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") "
Oct 30 22:12:02.075032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.074982    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\") pod \"5f1bf09f-42c7-4382-9eef-070316319ab5\" (UID: \"5f1bf09f-42c7-4382-9eef-070316319ab5\") "
Oct 30 22:12:02.075354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.075323    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume" pod "5f1bf09f-42c7-4382-9eef-070316319ab5" (UID: "5f1bf09f-42c7-4382-9eef-070316319ab5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:02.076858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.076837    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm" pod "5f1bf09f-42c7-4382-9eef-070316319ab5" (UID: "5f1bf09f-42c7-4382-9eef-070316319ab5"). InnerVolumeSpecName "kube-api-access-7s8cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:02.176878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.176843    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1bf09f-42c7-4382-9eef-070316319ab5-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:02.176878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.176876    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8cm\" (UniqueName: \"kubernetes.io/projected/5f1bf09f-42c7-4382-9eef-070316319ab5-kube-api-access-7s8cm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:02.945829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.945768    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"
Oct 30 22:12:02.945829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.945817    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9" event={"ID":"5f1bf09f-42c7-4382-9eef-070316319ab5","Type":"ContainerDied","Data":"78b76aaa3922b9c74a8f08d44af66f217f0efaefb6e6b5d30d2fccb03422302f"}
Oct 30 22:12:02.946413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:02.945881    4410 scope.go:122] "RemoveContainer" containerID="25a0b165baa76471724d28dfa1b74457496ef1c502e75d59986d72565d3531d9"
Oct 30 22:12:03.112552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:03.112519    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:12:03.115743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:03.115714    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-479363-d9446c767-kl9d9"]
Oct 30 22:12:03.338440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:03.338357    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:05.019107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:05.019073    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1bf09f-42c7-4382-9eef-070316319ab5" path="/var/lib/kubelet/pods/5f1bf09f-42c7-4382-9eef-070316319ab5/volumes"
Oct 30 22:12:08.238874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:08.238831    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:08.476787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:08.476745    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:13.550920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:13.550879    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:17.247612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:17.247570    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:17.892801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:17.891489    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.130373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:18.130341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.710646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:18.710605    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.523276 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:12:20.523238    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:12:21.049264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:21.049232    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"]
Oct 30 22:12:21.098628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:21.098597    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15815553-pfx42"]
Oct 30 22:12:22.925805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:22.925732    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.020462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:23.020419    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32dc8093-53fa-4afe-bbb7-dfc95af8dd52" path="/var/lib/kubelet/pods/32dc8093-53fa-4afe-bbb7-dfc95af8dd52/volumes"
Oct 30 22:12:23.182049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:23.181959    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.773366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:23.773308    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.943011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:23.942976    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.239285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:26.239254    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.435565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:26.435532    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.773510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:27.773479    4410 scope.go:122] "RemoveContainer" containerID="062f78f0ffd6cfcaf958984a6516d5d41c21c5546db291fd5f3745ac66965cad"
Oct 30 22:12:29.323832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:29.323795    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.052309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:32.052275    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"]
Oct 30 22:12:32.101284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:32.101253    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a5a1b8-9n79m"]
Oct 30 22:12:33.019388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:33.019352    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7c7ac8-28b9-4fa9-8a83-135500a06b63" path="/var/lib/kubelet/pods/4d7c7ac8-28b9-4fa9-8a83-135500a06b63/volumes"
Oct 30 22:12:34.380526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:34.380491    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.494811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:39.494757    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:46.058210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:46.058175    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.522144 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:12:50.522105    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:12:51.122678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:51.122639    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:54.586506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.586464    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:12:54.587320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587295    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2df2c45c-bca2-4a0d-a598-96bac3ec7db0" containerName="revoke-keys"
Oct 30 22:12:54.587320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587321    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df2c45c-bca2-4a0d-a598-96bac3ec7db0" containerName="revoke-keys"
Oct 30 22:12:54.587513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587348    4410 cpu_manager.go:414] "containerMap: removing container" podUID="5f1bf09f-42c7-4382-9eef-070316319ab5" containerName="payload-pre-processing"
Oct 30 22:12:54.587513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587358    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1bf09f-42c7-4382-9eef-070316319ab5" containerName="payload-pre-processing"
Oct 30 22:12:54.587513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587383    4410 cpu_manager.go:414] "containerMap: removing container" podUID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" containerName="payload-pre-processing"
Oct 30 22:12:54.587513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587391    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" containerName="payload-pre-processing"
Oct 30 22:12:54.587742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587544    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df2c45c-bca2-4a0d-a598-96bac3ec7db0" containerName="revoke-keys"
Oct 30 22:12:54.587742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587564    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1bf09f-42c7-4382-9eef-070316319ab5" containerName="payload-pre-processing"
Oct 30 22:12:54.587742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.587575    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d7fee0-5f25-405a-aa09-1f7d569c78b8" containerName="payload-pre-processing"
Oct 30 22:12:54.591389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.591367    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.703703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.703672    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.703916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.703708    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.703916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.703834    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.703937    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.703960    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.704033    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.704164    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.704206    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.704306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.704251    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.711467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.711443    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:12:54.713898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.713872    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7fabe91d-openshift-default-dockercfg-rt9jz\""
Oct 30 22:12:54.806043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806013    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806175    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806211    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806257    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806351    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806388    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806456    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806523    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806557    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806557    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806458    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.806993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806969    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.807154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806982    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.807270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.806982    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.809498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.809452    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.809748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.809728    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.867515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.867455    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.868145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.868124    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:54.906324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:54.906303    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:55.342218 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:12:55.342190    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340e5af2_0214_409e_aa6b_1d435860a223.slice/crio-386a128f2ad75903aaa269007d5393ec0b742d8890529f9610854395014bfeab WatchSource:0}: Error finding container 386a128f2ad75903aaa269007d5393ec0b742d8890529f9610854395014bfeab: Status 404 returned error can't find the container with id 386a128f2ad75903aaa269007d5393ec0b742d8890529f9610854395014bfeab
Oct 30 22:12:55.343463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:55.343439    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:12:55.344806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:55.344754    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:55.344865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:55.344839    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:55.344905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:55.344869    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:56.322598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.322549    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" event={"ID":"340e5af2-0214-409e-aa6b-1d435860a223","Type":"ContainerStarted","Data":"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"}
Oct 30 22:12:56.323112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.322625    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" event={"ID":"340e5af2-0214-409e-aa6b-1d435860a223","Type":"ContainerStarted","Data":"386a128f2ad75903aaa269007d5393ec0b742d8890529f9610854395014bfeab"}
Oct 30 22:12:56.333735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.333710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:56.907272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.907234    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:12:56.908434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.908412    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.908538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:56.908468    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 22:12:57.907482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:57.907451    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.907896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:57.907509    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 22:12:58.907208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:58.907175    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.907383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:58.907235    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 22:12:59.907357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:59.907321    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.907748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:12:59.907383    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 22:13:00.916054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:00.916020    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.916675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:00.916632    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 22:13:01.910466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:01.910428    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:02.193063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:02.192987    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:13:02.193941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:02.193912    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:13:02.195087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:02.195052    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:13:02.414689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:02.414635    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podStartSLOduration=8.414617997 podStartE2EDuration="8.414617997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:56.566073145 +0000 UTC m=+1829.654810951" watchObservedRunningTime="2026-10-30 22:13:02.414617997 +0000 UTC m=+1835.503355813"
Oct 30 22:13:04.328548 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:04.328514    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.602992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:06.602952    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:07.502961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.502912    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:13:07.503247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.503199    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" containerID="cri-o://45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9" gracePeriod=30
Oct 30 22:13:07.530087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.530049    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:13:07.530600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.530568    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" containerID="cri-o://923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e" gracePeriod=30
Oct 30 22:13:07.532846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.532822    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:13:07.533030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.533009    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy" containerID="cri-o://c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36" gracePeriod=30
Oct 30 22:13:07.563657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.563621    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:13:07.563899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:07.563875    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy" containerID="cri-o://31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf" gracePeriod=30
Oct 30 22:13:08.192147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:08.192110    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.443877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:08.443801    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.614423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:08.614382    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.669892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:08.669861    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"]
Oct 30 22:13:08.672728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:08.672708    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d73ad7-cn5qx"]
Oct 30 22:13:09.019985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.019954    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df2c45c-bca2-4a0d-a598-96bac3ec7db0" path="/var/lib/kubelet/pods/2df2c45c-bca2-4a0d-a598-96bac3ec7db0/volumes"
Oct 30 22:13:09.666574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.666537    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:13:09.670770 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.670750    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.781642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.781610    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm87w\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.781843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.781672    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.781843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.781802    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.781994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.781848    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.781994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.781972    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.782105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.782012    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.782216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.782190    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.782303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.782283    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.782385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.782361    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.879913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.879714    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:13:09.884134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884104    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884194    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884222    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884326    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884382    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884445    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884499    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm87w\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884566    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884575    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884610    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884623    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.884920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.884896    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.885080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.885061    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.885179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.885133    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.887341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.887315    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.887489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.887469    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.941169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.941104    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.941463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.941444    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm87w\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:09.984691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:09.984665    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:10.417649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:10.417619    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:13:10.418485 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:13:10.418460    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e25453d_3a27_4485_9bdf_fccee9c895bd.slice/crio-e01bdcba041bbfc53a4e060b5092e221180e62e819e030c0d6da0ba577b2552d WatchSource:0}: Error finding container e01bdcba041bbfc53a4e060b5092e221180e62e819e030c0d6da0ba577b2552d: Status 404 returned error can't find the container with id e01bdcba041bbfc53a4e060b5092e221180e62e819e030c0d6da0ba577b2552d
Oct 30 22:13:10.421027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:10.420996    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:10.421088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:10.421068    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:10.421130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:10.421098    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:10.482467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:10.482414    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" event={"ID":"2e25453d-3a27-4485-9bdf-fccee9c895bd","Type":"ContainerStarted","Data":"e01bdcba041bbfc53a4e060b5092e221180e62e819e030c0d6da0ba577b2552d"}
Oct 30 22:13:11.519631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.519592    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" event={"ID":"2e25453d-3a27-4485-9bdf-fccee9c895bd","Type":"ContainerStarted","Data":"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"}
Oct 30 22:13:11.740825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.740755    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podStartSLOduration=2.740741441 podStartE2EDuration="2.740741441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:11.690462157 +0000 UTC m=+1844.779199999" watchObservedRunningTime="2026-10-30 22:13:11.740741441 +0000 UTC m=+1844.829479244"
Oct 30 22:13:11.971237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.971203    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:13:11.971460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.971434    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:13:11.985721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.985698    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:11.986758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.986737    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:13:11.986880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:11.986787    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 22:13:12.531966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.531930    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:13:12.532340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.531986    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:13:12.941639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.941174    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:13:12.965802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.944384    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:13:12.986049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.985986    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:13:12.986169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.986067    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 22:13:12.993257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.993237    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:13:12.995697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:12.995679    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:13:13.034112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034089    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034174    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.034231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034203    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034222    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034244    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034287    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034339    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.034426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034379    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.034426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034384    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.034697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034479    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.034697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034480    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.034697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034499    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.034697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034684    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.034965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034922    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.035021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.034989    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:13.035083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035016    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.035083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035059    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.035195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035103    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.035195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035150    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.035195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035190    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.035345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035217    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.035345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035286    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.035453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035382    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\") pod \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\" (UID: \"36eb9dc3-6c57-42b7-850e-72c689bb3a05\") "
Oct 30 22:13:13.035453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035409    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.035453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.035414    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\") pod \"6785ae40-2209-4747-8b0d-387c81c481b2\" (UID: \"6785ae40-2209-4747-8b0d-387c81c481b2\") "
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036072    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036100    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036116    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036132    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036147    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036162    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.036369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036176    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.037167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.036896    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.037167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.037095    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "kube-api-access-jkx55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.037346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.037317    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.037406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.037381    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:13.037562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.037546    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.038028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.038006    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:13.038139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.038038    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "kube-api-access-52z9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.038863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.038835    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:13.039007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.038985    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token" pod "6785ae40-2209-4747-8b0d-387c81c481b2" (UID: "6785ae40-2209-4747-8b0d-387c81c481b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.039302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.039261    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.039484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.039462    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token" pod "36eb9dc3-6c57-42b7-850e-72c689bb3a05" (UID: "36eb9dc3-6c57-42b7-850e-72c689bb3a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.137195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137118    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137161    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137214    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137242    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137274    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137377    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137380    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.137432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137430    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137452    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137487    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137543    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137572    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137614    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137653    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137713    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.137856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137826    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137873    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\") pod \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\" (UID: \"92e6b31d-a8c1-4e62-81a9-d62d14c44932\") "
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137890    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137905    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137956    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\") pod \"b3420b78-7732-4efa-9fee-a3927a67b983\" (UID: \"b3420b78-7732-4efa-9fee-a3927a67b983\") "
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.137987    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:13.138280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138098    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.138586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138283    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.138586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138438    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.138819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138795    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785ae40-2209-4747-8b0d-387c81c481b2-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138825    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138840    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138856    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138870    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138886    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.138911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138901    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138914    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138930    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138944    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3420b78-7732-4efa-9fee-a3927a67b983-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138957    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138972    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.138985    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52z9l\" (UniqueName: \"kubernetes.io/projected/36eb9dc3-6c57-42b7-850e-72c689bb3a05-kube-api-access-52z9l\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139001    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139015    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkx55\" (UniqueName: \"kubernetes.io/projected/6785ae40-2209-4747-8b0d-387c81c481b2-kube-api-access-jkx55\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139029    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785ae40-2209-4747-8b0d-387c81c481b2-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139042    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eb9dc3-6c57-42b7-850e-72c689bb3a05-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139056    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785ae40-2209-4747-8b0d-387c81c481b2-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.139812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139314    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.139812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.139567    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.140067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.140049    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:13.140371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.140345    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.140805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.140743    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "kube-api-access-7lxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.141409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.141386    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "kube-api-access-cfhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.141752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.141735    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:13.141936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.141921    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:13.142422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.142401    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:13.142661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.142633    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy" pod "92e6b31d-a8c1-4e62-81a9-d62d14c44932" (UID: "92e6b31d-a8c1-4e62-81a9-d62d14c44932"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.143162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.143138    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy" pod "b3420b78-7732-4efa-9fee-a3927a67b983" (UID: "b3420b78-7732-4efa-9fee-a3927a67b983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:13.240371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240339    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3420b78-7732-4efa-9fee-a3927a67b983-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240373    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240386    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240399    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240413    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240430    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhn7\" (UniqueName: \"kubernetes.io/projected/92e6b31d-a8c1-4e62-81a9-d62d14c44932-kube-api-access-cfhn7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240445    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240461    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240477    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3420b78-7732-4efa-9fee-a3927a67b983-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240490    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxtm\" (UniqueName: \"kubernetes.io/projected/b3420b78-7732-4efa-9fee-a3927a67b983-kube-api-access-7lxtm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.240555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.240504    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6b31d-a8c1-4e62-81a9-d62d14c44932-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:13.535550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.535516    4410 generic.go:356] "Generic (PLEG): container finished" podID="b3420b78-7732-4efa-9fee-a3927a67b983" containerID="923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e" exitCode=0
Oct 30 22:13:13.536066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.535560    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"
Oct 30 22:13:13.536066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.535596    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" event={"ID":"b3420b78-7732-4efa-9fee-a3927a67b983","Type":"ContainerDied","Data":"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"}
Oct 30 22:13:13.536066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.535656    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk" event={"ID":"b3420b78-7732-4efa-9fee-a3927a67b983","Type":"ContainerDied","Data":"16a64adda670568226ce39c98aa5291d849174d76ffd10d4a862ab56b31124b9"}
Oct 30 22:13:13.536066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.535684    4410 scope.go:122] "RemoveContainer" containerID="923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"
Oct 30 22:13:13.537370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.537345    4410 generic.go:356] "Generic (PLEG): container finished" podID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerID="c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36" exitCode=0
Oct 30 22:13:13.537473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.537390    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"
Oct 30 22:13:13.537473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.537432    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" event={"ID":"92e6b31d-a8c1-4e62-81a9-d62d14c44932","Type":"ContainerDied","Data":"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"}
Oct 30 22:13:13.537573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.537483    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb" event={"ID":"92e6b31d-a8c1-4e62-81a9-d62d14c44932","Type":"ContainerDied","Data":"3393aed454e863b770f1b738091c9301a1aa4d8375d17c470472332c126021cd"}
Oct 30 22:13:13.539064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.539037    4410 generic.go:356] "Generic (PLEG): container finished" podID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerID="45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9" exitCode=0
Oct 30 22:13:13.539175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.539106    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"
Oct 30 22:13:13.539175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.539126    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" event={"ID":"36eb9dc3-6c57-42b7-850e-72c689bb3a05","Type":"ContainerDied","Data":"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"}
Oct 30 22:13:13.539287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.539175    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v" event={"ID":"36eb9dc3-6c57-42b7-850e-72c689bb3a05","Type":"ContainerDied","Data":"a555588a58ee2862bdab44a688d01b87c6ec75afc035a66f2e6d117e3fb8bdc9"}
Oct 30 22:13:13.541343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.541313    4410 generic.go:356] "Generic (PLEG): container finished" podID="6785ae40-2209-4747-8b0d-387c81c481b2" containerID="31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf" exitCode=0
Oct 30 22:13:13.541462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.541373    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"
Oct 30 22:13:13.541462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.541432    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" event={"ID":"6785ae40-2209-4747-8b0d-387c81c481b2","Type":"ContainerDied","Data":"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"}
Oct 30 22:13:13.541576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.541474    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg" event={"ID":"6785ae40-2209-4747-8b0d-387c81c481b2","Type":"ContainerDied","Data":"03ce133331fe1af8496b12104019e478633831b63b25e93b903663feaa680eac"}
Oct 30 22:13:13.542619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.542588    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:13.600024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.600000    4410 scope.go:122] "RemoveContainer" containerID="923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"
Oct 30 22:13:13.600431 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:13.600407    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e\": container with ID starting with 923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e not found: ID does not exist" containerID="923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"
Oct 30 22:13:13.600514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.600440    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e"} err="failed to get container status \"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e\": rpc error: code = NotFound desc = could not find container \"923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e\": container with ID starting with 923f4c005e33a6a643295a599700276af516578756d6c7f077947abdcbfbf86e not found: ID does not exist"
Oct 30 22:13:13.600514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.600456    4410 scope.go:122] "RemoveContainer" containerID="c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"
Oct 30 22:13:13.610846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.610823    4410 scope.go:122] "RemoveContainer" containerID="c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"
Oct 30 22:13:13.611306 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:13.611173    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36\": container with ID starting with c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36 not found: ID does not exist" containerID="c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"
Oct 30 22:13:13.611306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.611210    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36"} err="failed to get container status \"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36\": rpc error: code = NotFound desc = could not find container \"c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36\": container with ID starting with c09380cdae175902a6ce09a083beab84e1c6d5f3973a11d68a2b73f6aeb86c36 not found: ID does not exist"
Oct 30 22:13:13.611306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.611229    4410 scope.go:122] "RemoveContainer" containerID="45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"
Oct 30 22:13:13.629977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.629949    4410 scope.go:122] "RemoveContainer" containerID="45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"
Oct 30 22:13:13.630386 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:13.630351    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9\": container with ID starting with 45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9 not found: ID does not exist" containerID="45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"
Oct 30 22:13:13.630476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.630403    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9"} err="failed to get container status \"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9\": rpc error: code = NotFound desc = could not find container \"45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9\": container with ID starting with 45f9cdfdee458740afda1cc3c807999d38c510a5c97ec8c29df56c8d6a7da3e9 not found: ID does not exist"
Oct 30 22:13:13.630476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.630430    4410 scope.go:122] "RemoveContainer" containerID="31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"
Oct 30 22:13:13.641598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.641579    4410 scope.go:122] "RemoveContainer" containerID="31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"
Oct 30 22:13:13.641931 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:13.641909    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf\": container with ID starting with 31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf not found: ID does not exist" containerID="31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"
Oct 30 22:13:13.642019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.641941    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf"} err="failed to get container status \"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf\": rpc error: code = NotFound desc = could not find container \"31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf\": container with ID starting with 31dabc367e3de1b18718229752529d12ca8ee2d7804c35728997e2aaa132a6bf not found: ID does not exist"
Oct 30 22:13:13.765544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.760633    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:13.875470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.875379    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:13:13.878432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.878402    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-kxvbg"]
Oct 30 22:13:13.971398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.971367    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:13.985405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.985373    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:13:13.985644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.985619    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:13:13.985755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.985664    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 22:13:13.989861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:13.989838    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-j4s5v"]
Oct 30 22:13:14.091449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.091416    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"]
Oct 30 22:13:14.091983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.091968    4410 cpu_manager.go:414] "containerMap: removing container" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.091985    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092002    4410 cpu_manager.go:414] "containerMap: removing container" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092008    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092018    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092023    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092041    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy"
Oct 30 22:13:14.092044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092046    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy"
Oct 30 22:13:14.092253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092116    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" containerName="istio-proxy"
Oct 30 22:13:14.092253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092127    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" containerName="istio-proxy"
Oct 30 22:13:14.092253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092133    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" containerName="istio-proxy"
Oct 30 22:13:14.092253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.092143    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" containerName="istio-proxy"
Oct 30 22:13:14.097057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.097039    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.103274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.103251    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:13:14.109703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.109676    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-rnrsk"]
Oct 30 22:13:14.163795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.163171    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:13:14.163795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.163247    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:14.169800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.167633    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:14.169800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.167728    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:14.173509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.173483    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"]
Oct 30 22:13:14.229451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.229419    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:13:14.236194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.236153    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-md8vb"]
Oct 30 22:13:14.287022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.286993    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.287168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.287151    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crhn\" (UniqueName: \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.287325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.287310    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.388294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.388259    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crhn\" (UniqueName: \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.388462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.388395    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.388462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.388418    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.389092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.389066    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.390713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.390688    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.466617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.466543    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crhn\" (UniqueName: \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\") pod \"maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.714742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.714714    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:14.985665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.985586    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:13:14.985665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:14.985639    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 22:13:15.021047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.021013    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36eb9dc3-6c57-42b7-850e-72c689bb3a05" path="/var/lib/kubelet/pods/36eb9dc3-6c57-42b7-850e-72c689bb3a05/volumes"
Oct 30 22:13:15.021593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.021570    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6785ae40-2209-4747-8b0d-387c81c481b2" path="/var/lib/kubelet/pods/6785ae40-2209-4747-8b0d-387c81c481b2/volumes"
Oct 30 22:13:15.022127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.022106    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e6b31d-a8c1-4e62-81a9-d62d14c44932" path="/var/lib/kubelet/pods/92e6b31d-a8c1-4e62-81a9-d62d14c44932/volumes"
Oct 30 22:13:15.023118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.023093    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3420b78-7732-4efa-9fee-a3927a67b983" path="/var/lib/kubelet/pods/b3420b78-7732-4efa-9fee-a3927a67b983/volumes"
Oct 30 22:13:15.117204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.117177    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"]
Oct 30 22:13:15.118966 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:13:15.118935    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08618060_762d_4473_b738_f9c364f1a492.slice/crio-27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0 WatchSource:0}: Error finding container 27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0: Status 404 returned error can't find the container with id 27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0
Oct 30 22:13:15.587245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.587215    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr" event={"ID":"08618060-762d-4473-b738-f9c364f1a492","Type":"ContainerStarted","Data":"9fca73136863da1c77e2c799b6a1bf27ea7caec9c73142c5e4afb83b43ecfef6"}
Oct 30 22:13:15.587354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.587269    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr" event={"ID":"08618060-762d-4473-b738-f9c364f1a492","Type":"ContainerStarted","Data":"27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0"}
Oct 30 22:13:15.709601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.709547    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr" podStartSLOduration=1.7095345389999999 podStartE2EDuration="1.709534539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:15.706254713 +0000 UTC m=+1848.794992516" watchObservedRunningTime="2026-10-30 22:13:15.709534539 +0000 UTC m=+1848.798272344"
Oct 30 22:13:15.986077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.985972    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:13:15.986077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:15.986035    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 22:13:16.596122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:16.596031    4410 generic.go:356] "Generic (PLEG): container finished" podID="08618060-762d-4473-b738-f9c364f1a492" containerID="9fca73136863da1c77e2c799b6a1bf27ea7caec9c73142c5e4afb83b43ecfef6" exitCode=0
Oct 30 22:13:16.596281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:16.596112    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr" event={"ID":"08618060-762d-4473-b738-f9c364f1a492","Type":"ContainerDied","Data":"9fca73136863da1c77e2c799b6a1bf27ea7caec9c73142c5e4afb83b43ecfef6"}
Oct 30 22:13:16.989439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:16.989408    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:16.989859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:16.989673    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:16.990406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:16.990387    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:13:17.756407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.756383    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:17.836019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.835991    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\") pod \"08618060-762d-4473-b738-f9c364f1a492\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") "
Oct 30 22:13:17.836186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.836170    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\" (UniqueName: \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\") pod \"08618060-762d-4473-b738-f9c364f1a492\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") "
Oct 30 22:13:17.836279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.836267    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\") pod \"08618060-762d-4473-b738-f9c364f1a492\" (UID: \"08618060-762d-4473-b738-f9c364f1a492\") "
Oct 30 22:13:17.836343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.836309    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca" pod "08618060-762d-4473-b738-f9c364f1a492" (UID: "08618060-762d-4473-b738-f9c364f1a492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:17.836690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.836668    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08618060-762d-4473-b738-f9c364f1a492-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:17.838368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.838344    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn" pod "08618060-762d-4473-b738-f9c364f1a492" (UID: "08618060-762d-4473-b738-f9c364f1a492"). InnerVolumeSpecName "kube-api-access-8crhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:17.838458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.838415    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca" pod "08618060-762d-4473-b738-f9c364f1a492" (UID: "08618060-762d-4473-b738-f9c364f1a492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:17.937946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.937922    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crhn\" (UniqueName: \"kubernetes.io/projected/08618060-762d-4473-b738-f9c364f1a492-kube-api-access-8crhn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:17.937946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:17.937947    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08618060-762d-4473-b738-f9c364f1a492-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:18.611885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:18.611856    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"
Oct 30 22:13:18.612374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:18.611900    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr" event={"ID":"08618060-762d-4473-b738-f9c364f1a492","Type":"ContainerDied","Data":"27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0"}
Oct 30 22:13:18.612374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:18.611946    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ee35c29e2046665781600459e1af98638ed8a246105612f143077c2b8060a0"
Oct 30 22:13:19.329684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:19.329646    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:20.522016 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:20.521968    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:13:23.700577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:23.700545    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:24.056187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:24.056100    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"]
Oct 30 22:13:24.103805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:24.103758    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10b9da98-2rtqg"]
Oct 30 22:13:24.134337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:24.134309    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:24.376637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:24.376555    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:25.020212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.020178    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9e0f75-9fd0-40a0-8875-d79524eb395f" path="/var/lib/kubelet/pods/ff9e0f75-9fd0-40a0-8875-d79524eb395f/volumes"
Oct 30 22:13:25.286357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.286280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:25.754707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.754671    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:25.819228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.819193    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"]
Oct 30 22:13:25.868246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.868211    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02a8fa-74gdw"]
Oct 30 22:13:25.948092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:25.948059    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:26.157178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:26.157148    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:26.332186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:26.330362    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.019840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:27.019810    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f6abf8-a5ed-44b2-b34a-5377db608d81" path="/var/lib/kubelet/pods/75f6abf8-a5ed-44b2-b34a-5377db608d81/volumes"
Oct 30 22:13:27.810308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:27.810275    4410 scope.go:122] "RemoveContainer" containerID="586f8939f58e28b317430b1e9721eb49e990a2778a0b9dec90f017683ba08e6c"
Oct 30 22:13:27.820096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:27.820077    4410 scope.go:122] "RemoveContainer" containerID="7272e2674e2243a53fb559639c4003eb8e7226197160dc9414b51e42afed10aa"
Oct 30 22:13:29.083735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:29.083695    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.859961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:30.859925    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.324387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:32.324348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.539674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:32.539634    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.698195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:32.698165    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:33.629487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:33.629450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.275796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:34.262264    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.473981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:34.473946    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.713765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:34.713726    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:35.275607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:35.275571    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.580538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:36.580497    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.819389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:36.819350    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.908069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:36.908037    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:13:36.908347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:36.908317    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy" containerID="cri-o://47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e" gracePeriod=30
Oct 30 22:13:37.061828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.054471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.268982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.268896    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.433621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.433580    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.493056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.493024    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"]
Oct 30 22:13:37.495717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.495697    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-479363-csxj9"]
Oct 30 22:13:37.883800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.883749    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.892353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.892329    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"]
Oct 30 22:13:37.892856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.892843    4410 cpu_manager.go:414] "containerMap: removing container" podUID="08618060-762d-4473-b738-f9c364f1a492" containerName="revoke-keys"
Oct 30 22:13:37.892904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.892857    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08618060-762d-4473-b738-f9c364f1a492" containerName="revoke-keys"
Oct 30 22:13:37.892953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.892944    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="08618060-762d-4473-b738-f9c364f1a492" containerName="revoke-keys"
Oct 30 22:13:37.897986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.897964    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:37.929748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.929725    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:37.929969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.929952    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:37.930044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.930026    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlz45\" (UniqueName: \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:37.950936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.950419    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:37.950936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.950431    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:37.950936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.950706    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:37.950936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.950745    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:13:37.953145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:37.953121    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"]
Oct 30 22:13:38.030862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.030838    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.031022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.030912    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlz45\" (UniqueName: \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.031126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.031106    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.032388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.032361    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.033674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.033650    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.087821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.087795    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlz45\" (UniqueName: \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\") pod \"maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.211179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.211151    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:38.584891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.584860    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"]
Oct 30 22:13:38.586235 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:13:38.586204    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143d0e33_c342_4c92_bdce_126c99b65fcc.slice/crio-7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798 WatchSource:0}: Error finding container 7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798: Status 404 returned error can't find the container with id 7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798
Oct 30 22:13:38.907513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.907486    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b" event={"ID":"143d0e33-c342-4c92-bdce-126c99b65fcc","Type":"ContainerStarted","Data":"9fdb7aca99622f0354f17b26362748c2970c791e11f8c7eb61404c92d82a8e8c"}
Oct 30 22:13:38.907862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:38.907535    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b" event={"ID":"143d0e33-c342-4c92-bdce-126c99b65fcc","Type":"ContainerStarted","Data":"7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798"}
Oct 30 22:13:39.019162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.019134    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2539564f-af38-40a1-8617-ca6aafa70e62" path="/var/lib/kubelet/pods/2539564f-af38-40a1-8617-ca6aafa70e62/volumes"
Oct 30 22:13:39.026496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.026454    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b" podStartSLOduration=2.026442908 podStartE2EDuration="2.026442908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:39.022865635 +0000 UTC m=+1872.111603442" watchObservedRunningTime="2026-10-30 22:13:39.026442908 +0000 UTC m=+1872.115180724"
Oct 30 22:13:39.450962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.450916    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.665463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.665436    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.915514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.915480    4410 generic.go:356] "Generic (PLEG): container finished" podID="143d0e33-c342-4c92-bdce-126c99b65fcc" containerID="9fdb7aca99622f0354f17b26362748c2970c791e11f8c7eb61404c92d82a8e8c" exitCode=0
Oct 30 22:13:39.915964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:39.915567    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b" event={"ID":"143d0e33-c342-4c92-bdce-126c99b65fcc","Type":"ContainerDied","Data":"9fdb7aca99622f0354f17b26362748c2970c791e11f8c7eb61404c92d82a8e8c"}
Oct 30 22:13:41.079105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.079078    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:41.169573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.169540    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\") pod \"143d0e33-c342-4c92-bdce-126c99b65fcc\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") "
Oct 30 22:13:41.169813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.169770    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\") pod \"143d0e33-c342-4c92-bdce-126c99b65fcc\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") "
Oct 30 22:13:41.169897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.169879    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\" (UniqueName: \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\") pod \"143d0e33-c342-4c92-bdce-126c99b65fcc\" (UID: \"143d0e33-c342-4c92-bdce-126c99b65fcc\") "
Oct 30 22:13:41.170136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.170105    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca" pod "143d0e33-c342-4c92-bdce-126c99b65fcc" (UID: "143d0e33-c342-4c92-bdce-126c99b65fcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:41.170378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.170361    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:41.171669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.171642    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca" pod "143d0e33-c342-4c92-bdce-126c99b65fcc" (UID: "143d0e33-c342-4c92-bdce-126c99b65fcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:41.171907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.171890    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45" pod "143d0e33-c342-4c92-bdce-126c99b65fcc" (UID: "143d0e33-c342-4c92-bdce-126c99b65fcc"). InnerVolumeSpecName "kube-api-access-vlz45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:41.271436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.271401    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d0e33-c342-4c92-bdce-126c99b65fcc-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:41.271436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.271430    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlz45\" (UniqueName: \"kubernetes.io/projected/143d0e33-c342-4c92-bdce-126c99b65fcc-kube-api-access-vlz45\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:41.931477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.931451    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"
Oct 30 22:13:41.931599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.931454    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b" event={"ID":"143d0e33-c342-4c92-bdce-126c99b65fcc","Type":"ContainerDied","Data":"7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798"}
Oct 30 22:13:41.931599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:41.931581    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7396ba8af2babd2a233cfdb5d61c12c8a798afb50002588c2e264790fc413798"
Oct 30 22:13:42.195495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.195471    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:13:42.284507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284480    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284520    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284586    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284648    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284751    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:42.284850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284763    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284897    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.284970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284926    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:42.285010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.284973    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.285054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285012    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.285090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285079    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\") pod \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\" (UID: \"c1dc1069-d7d7-4f20-a895-54c4ebac2716\") "
Oct 30 22:13:42.285174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285153    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:42.285244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285076    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:42.285347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285316    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:42.285609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285589    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.285693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285615    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.285693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285630    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.285693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285642    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.285693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.285655    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.287133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.287113    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:42.287221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.287126    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:42.287272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.287236    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "kube-api-access-sv28t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:42.287330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.287314    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy" pod "c1dc1069-d7d7-4f20-a895-54c4ebac2716" (UID: "c1dc1069-d7d7-4f20-a895-54c4ebac2716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:42.386483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.386450    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.386483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.386478    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.386483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.386488    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv28t\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-kube-api-access-sv28t\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.386709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.386498    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dc1069-d7d7-4f20-a895-54c4ebac2716-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:42.729386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.729351    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:42.932825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.932789    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:42.938965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.938939    4410 generic.go:356] "Generic (PLEG): container finished" podID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerID="47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e" exitCode=0
Oct 30 22:13:42.939087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.938988    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"
Oct 30 22:13:42.939087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.939014    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerDied","Data":"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"}
Oct 30 22:13:42.939087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.939067    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4" event={"ID":"c1dc1069-d7d7-4f20-a895-54c4ebac2716","Type":"ContainerDied","Data":"505ff1ffd8c0f63f8157014b6b3e93901437056883e38c00f0c9b81d1a234860"}
Oct 30 22:13:42.939234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.939091    4410 scope.go:122] "RemoveContainer" containerID="47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"
Oct 30 22:13:42.949432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.949417    4410 scope.go:122] "RemoveContainer" containerID="190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"
Oct 30 22:13:42.958020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.958003    4410 scope.go:122] "RemoveContainer" containerID="47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"
Oct 30 22:13:42.958285 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:42.958264    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e\": container with ID starting with 47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e not found: ID does not exist" containerID="47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"
Oct 30 22:13:42.958372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.958291    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e"} err="failed to get container status \"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e\": rpc error: code = NotFound desc = could not find container \"47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e\": container with ID starting with 47ee58a7c248854ae2de2e2cacf08cf79961c0b315630295352d1a6251bac33e not found: ID does not exist"
Oct 30 22:13:42.958372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.958308    4410 scope.go:122] "RemoveContainer" containerID="190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"
Oct 30 22:13:42.958540 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:42.958521    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09\": container with ID starting with 190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09 not found: ID does not exist" containerID="190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"
Oct 30 22:13:42.958581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:42.958548    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09"} err="failed to get container status \"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09\": rpc error: code = NotFound desc = could not find container \"190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09\": container with ID starting with 190a2657d057d5d154ad0397e6534b9101ceea8320315c0fa3e813d79cd51e09 not found: ID does not exist"
Oct 30 22:13:43.111626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:43.111545    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:13:43.160922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:43.160890    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d73ad7-openshift-default-77fbb68489-qnfn4"]
Oct 30 22:13:44.698830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:44.698793    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:44.988091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:44.988014    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.024355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:45.024320    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" path="/var/lib/kubelet/pods/c1dc1069-d7d7-4f20-a895-54c4ebac2716/volumes"
Oct 30 22:13:45.184490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:45.184455    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.406798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:45.404384    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:46.253198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:46.253168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.393510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:47.393406    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.580631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:47.580596    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:50.214673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:50.214635    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:50.522229 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:50.522146    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:13:51.833743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:51.833709    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:13:51.946834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:51.946802    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:13:51.976166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:51.976134    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:52.197446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:52.197411    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:52.364194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:52.364023    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:55.070988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:55.070950    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:55.900871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:55.900836    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:56.172642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:56.172544    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:56.834883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:56.834838    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" podUID="777fdffa-a0d5-465d-b000-bbfe819a983c" containerName="payload-processing" containerID="cri-o://685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0" gracePeriod=25
Oct 30 22:13:56.853486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:56.853458    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:56.947477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:56.947440    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" podUID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" containerName="payload-pre-processing" containerID="cri-o://64581b9e2e4129bae5ae328133fd83d06213f4bd953091c40b0ede4aa6548b5a" gracePeriod=25
Oct 30 22:13:57.111210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.111172    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:57.115343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.115316    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:13:57.117274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.117248    4410 generic.go:356] "Generic (PLEG): container finished" podID="777fdffa-a0d5-465d-b000-bbfe819a983c" containerID="685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0" exitCode=0
Oct 30 22:13:57.117374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.117302    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"
Oct 30 22:13:57.117374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.117324    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" event={"ID":"777fdffa-a0d5-465d-b000-bbfe819a983c","Type":"ContainerDied","Data":"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"}
Oct 30 22:13:57.117483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.117385    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb" event={"ID":"777fdffa-a0d5-465d-b000-bbfe819a983c","Type":"ContainerDied","Data":"aecccebfbb7b6ca867c213afc24e39825b611c12318f029cbf0a30f18ed0ab59"}
Oct 30 22:13:57.117483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.117419    4410 scope.go:122] "RemoveContainer" containerID="685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"
Oct 30 22:13:57.119065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.119040    4410 generic.go:356] "Generic (PLEG): container finished" podID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" containerID="64581b9e2e4129bae5ae328133fd83d06213f4bd953091c40b0ede4aa6548b5a" exitCode=0
Oct 30 22:13:57.119183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.119112    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" event={"ID":"82214732-f8ab-433e-a9d6-eb93dd3fd8e3","Type":"ContainerDied","Data":"64581b9e2e4129bae5ae328133fd83d06213f4bd953091c40b0ede4aa6548b5a"}
Oct 30 22:13:57.132929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.131154    4410 scope.go:122] "RemoveContainer" containerID="685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"
Oct 30 22:13:57.132929 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:13:57.131546    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0\": container with ID starting with 685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0 not found: ID does not exist" containerID="685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"
Oct 30 22:13:57.132929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.131577    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0"} err="failed to get container status \"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0\": rpc error: code = NotFound desc = could not find container \"685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0\": container with ID starting with 685d0512ecdfde25825a8d84bfaff4125514d7bfa9cbb2985e6c53d0c1fb60f0 not found: ID does not exist"
Oct 30 22:13:57.132929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.132566    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\" (UniqueName: \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\") pod \"777fdffa-a0d5-465d-b000-bbfe819a983c\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") "
Oct 30 22:13:57.133453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.133419    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\") pod \"777fdffa-a0d5-465d-b000-bbfe819a983c\" (UID: \"777fdffa-a0d5-465d-b000-bbfe819a983c\") "
Oct 30 22:13:57.133864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.133841    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume" pod "777fdffa-a0d5-465d-b000-bbfe819a983c" (UID: "777fdffa-a0d5-465d-b000-bbfe819a983c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:57.134151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.134132    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777fdffa-a0d5-465d-b000-bbfe819a983c-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.138602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.138566    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5" pod "777fdffa-a0d5-465d-b000-bbfe819a983c" (UID: "777fdffa-a0d5-465d-b000-bbfe819a983c"). InnerVolumeSpecName "kube-api-access-frpm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.225928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.225901    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:13:57.235310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.235285    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\") pod \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") "
Oct 30 22:13:57.235476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.235448    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\" (UniqueName: \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\") pod \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\" (UID: \"82214732-f8ab-433e-a9d6-eb93dd3fd8e3\") "
Oct 30 22:13:57.235698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.235677    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume" pod "82214732-f8ab-433e-a9d6-eb93dd3fd8e3" (UID: "82214732-f8ab-433e-a9d6-eb93dd3fd8e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:57.235946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.235933    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpm5\" (UniqueName: \"kubernetes.io/projected/777fdffa-a0d5-465d-b000-bbfe819a983c-kube-api-access-frpm5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.235986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.235951    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.237402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.237381    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62" pod "82214732-f8ab-433e-a9d6-eb93dd3fd8e3" (UID: "82214732-f8ab-433e-a9d6-eb93dd3fd8e3"). InnerVolumeSpecName "kube-api-access-kmm62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.283020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.282978    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:57.292630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.292604    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"]
Oct 30 22:13:57.298205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.298178    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7fabe91d-6hpbr"]
Oct 30 22:13:57.337076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.337043    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmm62\" (UniqueName: \"kubernetes.io/projected/82214732-f8ab-433e-a9d6-eb93dd3fd8e3-kube-api-access-kmm62\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.539107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.539073    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:13:57.542531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.542504    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2bc5e3-7764484d9c-c4ddb"]
Oct 30 22:13:57.740337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:57.740252    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:58.134734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:58.134662    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"
Oct 30 22:13:58.134893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:58.134665    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96" event={"ID":"82214732-f8ab-433e-a9d6-eb93dd3fd8e3","Type":"ContainerDied","Data":"88ec795dc49ff6cab3b4baf56ed3d99f14b91e374a9e794a528c1e0f308b77c9"}
Oct 30 22:13:58.134893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:58.134855    4410 scope.go:122] "RemoveContainer" containerID="64581b9e2e4129bae5ae328133fd83d06213f4bd953091c40b0ede4aa6548b5a"
Oct 30 22:13:58.298355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:58.298325    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:13:58.347922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:58.347892    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2bc5e3-84ddddc6c-cvl96"]
Oct 30 22:13:59.019350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:59.019299    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08618060-762d-4473-b738-f9c364f1a492" path="/var/lib/kubelet/pods/08618060-762d-4473-b738-f9c364f1a492/volumes"
Oct 30 22:13:59.019900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:59.019878    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777fdffa-a0d5-465d-b000-bbfe819a983c" path="/var/lib/kubelet/pods/777fdffa-a0d5-465d-b000-bbfe819a983c/volumes"
Oct 30 22:13:59.020334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:13:59.020315    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" path="/var/lib/kubelet/pods/82214732-f8ab-433e-a9d6-eb93dd3fd8e3/volumes"
Oct 30 22:14:02.308626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:02.308580    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:02.667806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:02.667760    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:14:02.761529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:02.761493    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:06.656714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:06.656682    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:06.907096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:06.906990    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:07.812514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:07.812479    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:13.118008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:13.117963    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:17.688065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:17.688020    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" podUID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" containerName="main" containerID="cri-o://af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c" gracePeriod=45
Oct 30 22:14:17.957807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:17.957749    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:14:18.051281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.051246    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:18.087481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.087445    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.087870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.087833    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\" (UniqueName: \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.088163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.088088    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.088300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.088226    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.088300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.088267    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\" (UniqueName: \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.088418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.088388    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\") pod \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\" (UID: \"18145f3b-09f3-4a7e-9c2e-60ea8802742f\") "
Oct 30 22:14:18.089950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.089593    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.090317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.090283    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.090733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.090711    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.094083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.093891    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:18.094414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.094077    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "kube-api-access-8p48c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.099652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.099580    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm" pod "18145f3b-09f3-4a7e-9c2e-60ea8802742f" (UID: "18145f3b-09f3-4a7e-9c2e-60ea8802742f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190440    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190469    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190479    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p48c\" (UniqueName: \"kubernetes.io/projected/18145f3b-09f3-4a7e-9c2e-60ea8802742f-kube-api-access-8p48c\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190489    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190498    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18145f3b-09f3-4a7e-9c2e-60ea8802742f-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.190527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.190508    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18145f3b-09f3-4a7e-9c2e-60ea8802742f-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.256103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.256071    4410 generic.go:356] "Generic (PLEG): container finished" podID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" containerID="af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c" exitCode=0
Oct 30 22:14:18.256280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.256111    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"
Oct 30 22:14:18.256280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.256150    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" event={"ID":"18145f3b-09f3-4a7e-9c2e-60ea8802742f","Type":"ContainerDied","Data":"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"}
Oct 30 22:14:18.256280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.256204    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt" event={"ID":"18145f3b-09f3-4a7e-9c2e-60ea8802742f","Type":"ContainerDied","Data":"77055e02cc192214637440db9f26787afd4488558889d52c6e49147412ae78a6"}
Oct 30 22:14:18.256280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.256221    4410 scope.go:122] "RemoveContainer" containerID="af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"
Oct 30 22:14:18.266023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.266006    4410 scope.go:122] "RemoveContainer" containerID="af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"
Oct 30 22:14:18.266311 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:18.266291    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c\": container with ID starting with af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c not found: ID does not exist" containerID="af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"
Oct 30 22:14:18.266353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.266322    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c"} err="failed to get container status \"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c\": rpc error: code = NotFound desc = could not find container \"af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c\": container with ID starting with af13a4efddeaa087887b769c1c3dc7f91995acc166202ad6f1c92d6ef22ed92c not found: ID does not exist"
Oct 30 22:14:18.431030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.430989    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:14:18.443174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:18.443103    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7fabe91d/e2e-lifecycle-model-7fabe91d-kserve-6d5f8c4bfc-hz5lt"]
Oct 30 22:14:19.020303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:19.020265    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" path="/var/lib/kubelet/pods/18145f3b-09f3-4a7e-9c2e-60ea8802742f/volumes"
Oct 30 22:14:20.521646 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:20.521604    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:14:20.530470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:20.530440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:21.964305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:21.964272    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.121116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:22.121071    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.775002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:22.774963    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.989243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:22.989207    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.142696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:23.142662    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:24.466463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:24.466425    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:27.873967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:27.873934    4410 scope.go:122] "RemoveContainer" containerID="d46de72842dc3c9af196054c932f8d0e40012ee28b58922820d33a22c104b4f7"
Oct 30 22:14:27.885697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:27.885675    4410 scope.go:122] "RemoveContainer" containerID="0c08584a083d7f508dedc8922e8f5b97f58087f055c3cc6c3485a979bd54c9d1"
Oct 30 22:14:27.984592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:27.984563    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:34.429273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:34.429241    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:37.255098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:37.255065    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:14:37.255587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:37.255276    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" containerID="cri-o://145ce53da28465145b0de5e546d52a338c541e3180c3fca003cb8fab543c7900" gracePeriod=30
Oct 30 22:14:39.194259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:39.194220    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:39.363082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:39.363042    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:39.517245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:39.517145    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.333911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:40.333873    4410 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:40.334321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:40.333942    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:40.391024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:40.390992    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.840570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:40.840537    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:42.199965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.199933    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:14:42.200415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.200133    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" containerID="cri-o://0c24db8e45d5fe164eb159be46497cd3f0372c7cb5a35e35e60d000c79da8bd9" gracePeriod=30
Oct 30 22:14:42.249888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.249859    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:14:42.250130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.250091    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" containerID="cri-o://196d78c663104d68b6c22a15f85c276f759516e713006eb0248f81906c35702d" gracePeriod=30
Oct 30 22:14:42.401015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.400982    4410 generic.go:356] "Generic (PLEG): container finished" podID="780eaa75-630a-4d26-8b15-91e18de95b26" containerID="145ce53da28465145b0de5e546d52a338c541e3180c3fca003cb8fab543c7900" exitCode=0
Oct 30 22:14:42.401168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.401058    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" event={"ID":"780eaa75-630a-4d26-8b15-91e18de95b26","Type":"ContainerDied","Data":"145ce53da28465145b0de5e546d52a338c541e3180c3fca003cb8fab543c7900"}
Oct 30 22:14:42.523109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.523085    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:14:42.594524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594492    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594551    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594593    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594675    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594739    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594856    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594895    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.594947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594904    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:42.595097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594952    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.595097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.594991    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\") pod \"780eaa75-630a-4d26-8b15-91e18de95b26\" (UID: \"780eaa75-630a-4d26-8b15-91e18de95b26\") "
Oct 30 22:14:42.595097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595022    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:42.595248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595096    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:42.595248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595111    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:42.595248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595154    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:42.595565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595547    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.595654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595568    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.595654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595584    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780eaa75-630a-4d26-8b15-91e18de95b26-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.595654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595598    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.595654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.595612    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.597002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.596983    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:42.597077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.597002    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:42.597210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.597188    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:42.597256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.597222    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq" pod "780eaa75-630a-4d26-8b15-91e18de95b26" (UID: "780eaa75-630a-4d26-8b15-91e18de95b26"). InnerVolumeSpecName "kube-api-access-r5tjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:42.696821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.696791    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tjq\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-kube-api-access-r5tjq\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.696821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.696820    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780eaa75-630a-4d26-8b15-91e18de95b26-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.696997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.696831    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780eaa75-630a-4d26-8b15-91e18de95b26-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:42.696997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:42.696841    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780eaa75-630a-4d26-8b15-91e18de95b26-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:43.408447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:43.408410    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"
Oct 30 22:14:43.408993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:43.408438    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk" event={"ID":"780eaa75-630a-4d26-8b15-91e18de95b26","Type":"ContainerDied","Data":"029156284e154a63f390f9cd849273efcf197a8deb79d13cfacee2b9fb978858"}
Oct 30 22:14:43.408993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:43.408536    4410 scope.go:122] "RemoveContainer" containerID="145ce53da28465145b0de5e546d52a338c541e3180c3fca003cb8fab543c7900"
Oct 30 22:14:43.573160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:43.573098    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:14:43.622482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:43.622436    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-5klsk"]
Oct 30 22:14:44.276507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:44.276471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:45.020800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.020757    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" path="/var/lib/kubelet/pods/780eaa75-630a-4d26-8b15-91e18de95b26/volumes"
Oct 30 22:14:45.040328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.040301    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 22:14:45.040493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.040474    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" containerID="cri-o://39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951" gracePeriod=30
Oct 30 22:14:45.496652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.496615    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:45.496846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.496678    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:45.616726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.616694    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:45.616911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.616758    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:45.750101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.750028    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:45.750101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:45.750091    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:46.712901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:46.712863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:46.892964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:46.892927    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:47.038421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.038337    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:14:47.038990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.038909    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" containerID="cri-o://d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df" gracePeriod=30
Oct 30 22:14:47.046657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.046630    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:14:47.046850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.046816    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy" containerID="cri-o://ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263" gracePeriod=30
Oct 30 22:14:47.060169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.060126    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:47.196912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.196870    4410 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:47.197072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.196940    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:47.438083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.437888    4410 generic.go:356] "Generic (PLEG): container finished" podID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerID="0c24db8e45d5fe164eb159be46497cd3f0372c7cb5a35e35e60d000c79da8bd9" exitCode=0
Oct 30 22:14:47.438083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.438017    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" event={"ID":"708c45d4-5476-4f69-8449-121ffc27c1f7","Type":"ContainerDied","Data":"0c24db8e45d5fe164eb159be46497cd3f0372c7cb5a35e35e60d000c79da8bd9"}
Oct 30 22:14:47.441233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.441206    4410 generic.go:356] "Generic (PLEG): container finished" podID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerID="196d78c663104d68b6c22a15f85c276f759516e713006eb0248f81906c35702d" exitCode=0
Oct 30 22:14:47.441387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.441285    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" event={"ID":"1d65ac8b-c556-4dc0-bdb4-317d214bc465","Type":"ContainerDied","Data":"196d78c663104d68b6c22a15f85c276f759516e713006eb0248f81906c35702d"}
Oct 30 22:14:47.521874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.521843    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:14:47.555345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.555319    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:14:47.557484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557462    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557505    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557563    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557587    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557673    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557728    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557843    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.557949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557922    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.558002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557952    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.558002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557959    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:47.558002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.557993    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\") pod \"708c45d4-5476-4f69-8449-121ffc27c1f7\" (UID: \"708c45d4-5476-4f69-8449-121ffc27c1f7\") "
Oct 30 22:14:47.558196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558091    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.558248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558176    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.558248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558233    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.558546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558525    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708c45d4-5476-4f69-8449-121ffc27c1f7-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.558602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558554    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.558602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558571    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.558602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558584    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.558602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.558598    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.560075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.560047    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "kube-api-access-8cjnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:47.560075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.560036    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.560263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.560152    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:47.560489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.560475    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo" pod "708c45d4-5476-4f69-8449-121ffc27c1f7" (UID: "708c45d4-5476-4f69-8449-121ffc27c1f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:47.658986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.658953    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659004    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659067    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659105    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659130    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659161    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659256    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659327    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659355    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\") pod \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\" (UID: \"1d65ac8b-c556-4dc0-bdb4-317d214bc465\") "
Oct 30 22:14:47.659566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659402    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.659614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659583    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.659665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659615    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.659836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.659815    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660024    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660054    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660069    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660082    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660097    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660111    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjnn\" (UniqueName: \"kubernetes.io/projected/708c45d4-5476-4f69-8449-121ffc27c1f7-kube-api-access-8cjnn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660126    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708c45d4-5476-4f69-8449-121ffc27c1f7-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660139    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.660686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.660230    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:47.661704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.661670    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:47.662049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.662022    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:47.662209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.662190    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:47.662436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.662420    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg" pod "1d65ac8b-c556-4dc0-bdb4-317d214bc465" (UID: "1d65ac8b-c556-4dc0-bdb4-317d214bc465"). InnerVolumeSpecName "kube-api-access-w2ghg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:47.760992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.760962    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.760992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.760989    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.760992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.760999    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.761435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.761008    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ghg\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-kube-api-access-w2ghg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:47.761435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:47.761018    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d65ac8b-c556-4dc0-bdb4-317d214bc465-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:48.448219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.448192    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"
Oct 30 22:14:48.448382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.448235    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j" event={"ID":"1d65ac8b-c556-4dc0-bdb4-317d214bc465","Type":"ContainerDied","Data":"154e0f77789e0ea26d1ba27d1fc0e6ee8e943c1c283fc69e2ccf143d73637aad"}
Oct 30 22:14:48.448382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.448302    4410 scope.go:122] "RemoveContainer" containerID="196d78c663104d68b6c22a15f85c276f759516e713006eb0248f81906c35702d"
Oct 30 22:14:48.449822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.449802    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"
Oct 30 22:14:48.449944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.449803    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz" event={"ID":"708c45d4-5476-4f69-8449-121ffc27c1f7","Type":"ContainerDied","Data":"65d6dfa66da5cbc4b9d69720cfd7a37166edcbbadeaa12978b71568c46fbc398"}
Oct 30 22:14:48.457767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.457746    4410 scope.go:122] "RemoveContainer" containerID="0c24db8e45d5fe164eb159be46497cd3f0372c7cb5a35e35e60d000c79da8bd9"
Oct 30 22:14:48.593551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.593461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:48.614335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.614308    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:14:48.618347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.618320    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-bt2kz"]
Oct 30 22:14:48.728411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.728377    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:14:48.732522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:48.732495    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-bf85j"]
Oct 30 22:14:49.019322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:49.019280    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" path="/var/lib/kubelet/pods/1d65ac8b-c556-4dc0-bdb4-317d214bc465/volumes"
Oct 30 22:14:49.019999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:49.019976    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" path="/var/lib/kubelet/pods/708c45d4-5476-4f69-8449-121ffc27c1f7/volumes"
Oct 30 22:14:50.137410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.137378    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.289939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.289916    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 22:14:50.364807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.364726    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.390586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390560    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.390715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390609    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.390715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390693    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.390865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390798    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.390932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390896    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:50.391009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390913    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.391009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.390993    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.391191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391024    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.391191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391044    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.391191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391126    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\") pod \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\" (UID: \"6925356d-9e1e-44ef-9635-d6c9ad0ed416\") "
Oct 30 22:14:50.391356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391262    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:50.391356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391306    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:50.391485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391314    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:50.391485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391468    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:50.391795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391734    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.391921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391798    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.391921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391814    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.391921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391829    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.391921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.391843    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.393004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.392982    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "kube-api-access-5wzc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:50.393090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.393062    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:50.393768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.393742    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:50.393872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.393748    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo" pod "6925356d-9e1e-44ef-9635-d6c9ad0ed416" (UID: "6925356d-9e1e-44ef-9635-d6c9ad0ed416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:50.480492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.480454    4410 generic.go:356] "Generic (PLEG): container finished" podID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerID="39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951" exitCode=0
Oct 30 22:14:50.480694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.480517    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"
Oct 30 22:14:50.480694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.480522    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" event={"ID":"6925356d-9e1e-44ef-9635-d6c9ad0ed416","Type":"ContainerDied","Data":"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"}
Oct 30 22:14:50.480694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.480651    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn" event={"ID":"6925356d-9e1e-44ef-9635-d6c9ad0ed416","Type":"ContainerDied","Data":"9fea44081d97b148036bdc0fff627be696fac521a8f73855ab97a2cbf218250e"}
Oct 30 22:14:50.480694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.480682    4410 scope.go:122] "RemoveContainer" containerID="39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"
Oct 30 22:14:50.491070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.491050    4410 scope.go:122] "RemoveContainer" containerID="39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"
Oct 30 22:14:50.491374 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:50.491348    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951\": container with ID starting with 39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951 not found: ID does not exist" containerID="39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"
Oct 30 22:14:50.491446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.491388    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951"} err="failed to get container status \"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951\": rpc error: code = NotFound desc = could not find container \"39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951\": container with ID starting with 39e233c9b06c684fa8454aebb5c5954be40809ff75b23ca623472260862dc951 not found: ID does not exist"
Oct 30 22:14:50.493301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.493278    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.493357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.493310    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.493357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.493326    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzc4\" (UniqueName: \"kubernetes.io/projected/6925356d-9e1e-44ef-9635-d6c9ad0ed416-kube-api-access-5wzc4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.493357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.493340    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925356d-9e1e-44ef-9635-d6c9ad0ed416-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:50.521768 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:50.521738    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:14:50.553160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.553135    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.645601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.645572    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 22:14:50.647825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.647804    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-g5wxn"]
Oct 30 22:14:50.650319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.650300    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"]
Oct 30 22:14:50.693615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:50.693594    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2bc5e3-pnq6b"]
Oct 30 22:14:51.020136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:51.020058    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143d0e33-c342-4c92-bdce-126c99b65fcc" path="/var/lib/kubelet/pods/143d0e33-c342-4c92-bdce-126c99b65fcc/volumes"
Oct 30 22:14:51.020540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:51.020525    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" path="/var/lib/kubelet/pods/6925356d-9e1e-44ef-9635-d6c9ad0ed416/volumes"
Oct 30 22:14:51.079830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:51.079793    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:52.316569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.316516    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:14:52.341385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.341360    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:14:52.416529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416502    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416550    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.416686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416597    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416637    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416686    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416721    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416762    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.416894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416815    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.416894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416882    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416927    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416968    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417013    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417035    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\") pod \"340e5af2-0214-409e-aa6b-1d435860a223\" (UID: \"340e5af2-0214-409e-aa6b-1d435860a223\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417076    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417133    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417168    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417196    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417221    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\") pod \"2e25453d-3a27-4485-9bdf-fccee9c895bd\" (UID: \"2e25453d-3a27-4485-9bdf-fccee9c895bd\") "
Oct 30 22:14:52.417900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416921    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417955    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417968    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.416939    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417219    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417420    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417721    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.417839    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.418371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.418291    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:52.419013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.418971    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:52.419909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.419881    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.420362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.420319    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "kube-api-access-plmh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.420461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.420400    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.420461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.420399    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "kube-api-access-wm87w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.420581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.420485    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:52.421389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.421362    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:52.422223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.422199    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token" pod "2e25453d-3a27-4485-9bdf-fccee9c895bd" (UID: "2e25453d-3a27-4485-9bdf-fccee9c895bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.422311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.422292    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy" pod "340e5af2-0214-409e-aa6b-1d435860a223" (UID: "340e5af2-0214-409e-aa6b-1d435860a223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.497372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.497337    4410 generic.go:356] "Generic (PLEG): container finished" podID="340e5af2-0214-409e-aa6b-1d435860a223" containerID="d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df" exitCode=0
Oct 30 22:14:52.497505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.497380    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"
Oct 30 22:14:52.497505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.497414    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" event={"ID":"340e5af2-0214-409e-aa6b-1d435860a223","Type":"ContainerDied","Data":"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"}
Oct 30 22:14:52.497505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.497482    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c" event={"ID":"340e5af2-0214-409e-aa6b-1d435860a223","Type":"ContainerDied","Data":"386a128f2ad75903aaa269007d5393ec0b742d8890529f9610854395014bfeab"}
Oct 30 22:14:52.497664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.497514    4410 scope.go:122] "RemoveContainer" containerID="d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"
Oct 30 22:14:52.499467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.499441    4410 generic.go:356] "Generic (PLEG): container finished" podID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerID="ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263" exitCode=0
Oct 30 22:14:52.499568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.499501    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"
Oct 30 22:14:52.499568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.499521    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" event={"ID":"2e25453d-3a27-4485-9bdf-fccee9c895bd","Type":"ContainerDied","Data":"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"}
Oct 30 22:14:52.499685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.499583    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb" event={"ID":"2e25453d-3a27-4485-9bdf-fccee9c895bd","Type":"ContainerDied","Data":"e01bdcba041bbfc53a4e060b5092e221180e62e819e030c0d6da0ba577b2552d"}
Oct 30 22:14:52.507939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.507921    4410 scope.go:122] "RemoveContainer" containerID="d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"
Oct 30 22:14:52.508250 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:52.508203    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df\": container with ID starting with d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df not found: ID does not exist" containerID="d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"
Oct 30 22:14:52.508313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.508261    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df"} err="failed to get container status \"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df\": rpc error: code = NotFound desc = could not find container \"d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df\": container with ID starting with d8a6aa8d45cde6bfe4f200aaf2f4fa664f53c9d631950b31023c949999e131df not found: ID does not exist"
Oct 30 22:14:52.508313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.508282    4410 scope.go:122] "RemoveContainer" containerID="ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"
Oct 30 22:14:52.518787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518757    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.518861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518797    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.518861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518824    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340e5af2-0214-409e-aa6b-1d435860a223-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.518861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518839    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.518861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518851    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.518861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518859    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518867    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340e5af2-0214-409e-aa6b-1d435860a223-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518875    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518883    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518891    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518899    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340e5af2-0214-409e-aa6b-1d435860a223-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518907    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518917    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm87w\" (UniqueName: \"kubernetes.io/projected/2e25453d-3a27-4485-9bdf-fccee9c895bd-kube-api-access-wm87w\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518926    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518934    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e25453d-3a27-4485-9bdf-fccee9c895bd-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518944    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518952    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmh5\" (UniqueName: \"kubernetes.io/projected/340e5af2-0214-409e-aa6b-1d435860a223-kube-api-access-plmh5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.518961    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e25453d-3a27-4485-9bdf-fccee9c895bd-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.519564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.519547    4410 scope.go:122] "RemoveContainer" containerID="ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"
Oct 30 22:14:52.519853 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:14:52.519834    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263\": container with ID starting with ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263 not found: ID does not exist" containerID="ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"
Oct 30 22:14:52.519913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.519860    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263"} err="failed to get container status \"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263\": rpc error: code = NotFound desc = could not find container \"ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263\": container with ID starting with ca03a2e46445da64e6f12be00817137f95812bb6cdd5f3472bf989fec89ce263 not found: ID does not exist"
Oct 30 22:14:52.707893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.707854    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:14:52.711226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.711202    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-bn65c"]
Oct 30 22:14:52.818742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.818707    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:14:52.822937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:52.822907    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-tmdqb"]
Oct 30 22:14:53.020230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:53.020158    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" path="/var/lib/kubelet/pods/2e25453d-3a27-4485-9bdf-fccee9c895bd/volumes"
Oct 30 22:14:53.020599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:53.020579    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340e5af2-0214-409e-aa6b-1d435860a223" path="/var/lib/kubelet/pods/340e5af2-0214-409e-aa6b-1d435860a223/volumes"
Oct 30 22:14:55.771704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:55.771676    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:57.564531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:57.564495    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:57.794892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:57.794856    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:58.198197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:58.198141    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:58.376360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:58.376311    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:59.553948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:14:59.553907    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.072644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.072611    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.183013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.182977    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:00.183597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183570    4410 cpu_manager.go:414] "containerMap: removing container" podUID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" containerName="payload-pre-processing"
Oct 30 22:15:00.183597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183593    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" containerName="payload-pre-processing"
Oct 30 22:15:00.183597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183605    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.183597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183611    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183622    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183627    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183637    4410 cpu_manager.go:414] "containerMap: removing container" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183642    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183656    4410 cpu_manager.go:414] "containerMap: removing container" podUID="777fdffa-a0d5-465d-b000-bbfe819a983c" containerName="payload-processing"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183664    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777fdffa-a0d5-465d-b000-bbfe819a983c" containerName="payload-processing"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183676    4410 cpu_manager.go:414] "containerMap: removing container" podUID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" containerName="main"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183681    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" containerName="main"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183688    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183693    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183706    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183711    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183723    4410 cpu_manager.go:414] "containerMap: removing container" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183727    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183737    4410 cpu_manager.go:414] "containerMap: removing container" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183745    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183755    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183760    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183769    4410 cpu_manager.go:414] "containerMap: removing container" podUID="143d0e33-c342-4c92-bdce-126c99b65fcc" containerName="revoke-keys"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183792    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143d0e33-c342-4c92-bdce-126c99b65fcc" containerName="revoke-keys"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183877    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="780eaa75-630a-4d26-8b15-91e18de95b26" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183886    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183893    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="82214732-f8ab-433e-a9d6-eb93dd3fd8e3" containerName="payload-pre-processing"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183901    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e25453d-3a27-4485-9bdf-fccee9c895bd" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183909    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6925356d-9e1e-44ef-9635-d6c9ad0ed416" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183915    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="340e5af2-0214-409e-aa6b-1d435860a223" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183921    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="143d0e33-c342-4c92-bdce-126c99b65fcc" containerName="revoke-keys"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183928    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="777fdffa-a0d5-465d-b000-bbfe819a983c" containerName="payload-processing"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183935    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="18145f3b-09f3-4a7e-9c2e-60ea8802742f" containerName="main"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183948    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d65ac8b-c556-4dc0-bdb4-317d214bc465" containerName="istio-proxy"
Oct 30 22:15:00.183992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.183958    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="708c45d4-5476-4f69-8449-121ffc27c1f7" containerName="istio-proxy"
Oct 30 22:15:00.189181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.189157    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:00.189310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.189278    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.190455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.189898    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dc1069-d7d7-4f20-a895-54c4ebac2716" containerName="istio-proxy"
Oct 30 22:15:00.193420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.193399    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:00.193523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.193507    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.197276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.197256    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:00.197398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.197372    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.201732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.201712    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:00.201892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.201873    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.205282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.205264    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.244329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.244310    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.244454 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.244434    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.244547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.244534    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:15:00.256953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.256915    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:00.260860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.260843    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.280675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.280645    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.295205    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.299647    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304185    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mnc\" (UniqueName: \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304289    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wrz\" (UniqueName: \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304322    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304462    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304509    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304550    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xdm\" (UniqueName: \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304592    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzff\" (UniqueName: \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304621    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304668    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.304804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.304726    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkqn\" (UniqueName: \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.315525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.314628    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:00.396455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.396424    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:00.399205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.399176    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:00.406402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406369    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.406547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406439    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.406547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406499    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkqn\" (UniqueName: \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.406547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406539    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4g6\" (UniqueName: \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.406720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406588    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mnc\" (UniqueName: \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.406720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406697    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wrz\" (UniqueName: \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.406862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406744    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.407006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406867    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.407006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406937    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.407006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.406987    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.407179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407026    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xdm\" (UniqueName: \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.407179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407064    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzff\" (UniqueName: \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.407285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407260    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.407285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407269    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.407610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407585    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.407744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407721    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.407896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.407876    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.466372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.466313    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mnc\" (UniqueName: \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.466553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.466401    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wrz\" (UniqueName: \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.466729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.466687    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzff\" (UniqueName: \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.467410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.467387    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xdm\" (UniqueName: \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\") pod \"maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.468600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.468573    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkqn\" (UniqueName: \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.509267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.507951    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:00.511116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.511067    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4g6\" (UniqueName: \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.511642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.511605    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.512602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.512482    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.517863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.516930    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:00.523987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.523967    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:00.530925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.530896    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:00.537260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.537226    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:00.570196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.569968    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4g6\" (UniqueName: \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\") pod \"maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.573833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.573473    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:00.928451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.928418    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:00.930120 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:00.930094    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aabec7_5d5f_40a4_8b04_fd488eec2bcc.slice/crio-9b6661e43cfd016b76a803e4be0b2c9c6f083dcdf29c3d195a2a82afa8e991c7 WatchSource:0}: Error finding container 9b6661e43cfd016b76a803e4be0b2c9c6f083dcdf29c3d195a2a82afa8e991c7: Status 404 returned error can't find the container with id 9b6661e43cfd016b76a803e4be0b2c9c6f083dcdf29c3d195a2a82afa8e991c7
Oct 30 22:15:00.977817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.977756    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:00.978918 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:00.978866    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f143630_c95d_4376_8fde_5ab244b722c3.slice/crio-d2fc09babcbf805c3f999ecb0130c77f58324eb8b2e119b3c0a542b5bd5444fd WatchSource:0}: Error finding container d2fc09babcbf805c3f999ecb0130c77f58324eb8b2e119b3c0a542b5bd5444fd: Status 404 returned error can't find the container with id d2fc09babcbf805c3f999ecb0130c77f58324eb8b2e119b3c0a542b5bd5444fd
Oct 30 22:15:00.980291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:00.980273    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:00.980561 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:00.980524    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e22984_492a_4fe3_a47a_2fe15f5e06e1.slice/crio-be2e4a8d520e1d770d4fa699dc6d3df5f2e2db5730b1aaacea9ed9a12342033a WatchSource:0}: Error finding container be2e4a8d520e1d770d4fa699dc6d3df5f2e2db5730b1aaacea9ed9a12342033a: Status 404 returned error can't find the container with id be2e4a8d520e1d770d4fa699dc6d3df5f2e2db5730b1aaacea9ed9a12342033a
Oct 30 22:15:01.018422 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:01.017037    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d087e_1c88_4a16_874f_01c77c4a46ac.slice/crio-5be1110283511b1844fef4fcbe70c10ff1efc8f8e607c5ed25842257055170f0 WatchSource:0}: Error finding container 5be1110283511b1844fef4fcbe70c10ff1efc8f8e607c5ed25842257055170f0: Status 404 returned error can't find the container with id 5be1110283511b1844fef4fcbe70c10ff1efc8f8e607c5ed25842257055170f0
Oct 30 22:15:01.020935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.020912    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:01.033870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.033849    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:01.035532 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:01.035511    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e81557_79fc_44b4_85df_69b4e259e9ea.slice/crio-5fa3ffb06efeec7369f83361c7116e66683d47a15b917eaa65e797689ee6fccb WatchSource:0}: Error finding container 5fa3ffb06efeec7369f83361c7116e66683d47a15b917eaa65e797689ee6fccb: Status 404 returned error can't find the container with id 5fa3ffb06efeec7369f83361c7116e66683d47a15b917eaa65e797689ee6fccb
Oct 30 22:15:01.083094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.083070    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:01.148597 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:01.148495    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6502b9cd_a6c8_4fd8_9753_6c6bbc361b3c.slice/crio-95cc8bacd1526cbf3cdaca4923a74ed6393b46557c1cbf95c78be14bf1b18fe4 WatchSource:0}: Error finding container 95cc8bacd1526cbf3cdaca4923a74ed6393b46557c1cbf95c78be14bf1b18fe4: Status 404 returned error can't find the container with id 95cc8bacd1526cbf3cdaca4923a74ed6393b46557c1cbf95c78be14bf1b18fe4
Oct 30 22:15:01.557125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.557092    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerStarted","Data":"97e39fa4007a3b80d382442e3c3adb05639e4941544be4b88180848634d20916"}
Oct 30 22:15:01.557256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.557180    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerStarted","Data":"d2fc09babcbf805c3f999ecb0130c77f58324eb8b2e119b3c0a542b5bd5444fd"}
Oct 30 22:15:01.558986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.558959    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerStarted","Data":"5fa3ffb06efeec7369f83361c7116e66683d47a15b917eaa65e797689ee6fccb"}
Oct 30 22:15:01.560954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.560926    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerStarted","Data":"387b5d339b4377d6e97658fc1b7e42e396eac8d2df55221c5216df7dd8c18706"}
Oct 30 22:15:01.561072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.560976    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerStarted","Data":"be2e4a8d520e1d770d4fa699dc6d3df5f2e2db5730b1aaacea9ed9a12342033a"}
Oct 30 22:15:01.562210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.562160    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerStarted","Data":"95cc8bacd1526cbf3cdaca4923a74ed6393b46557c1cbf95c78be14bf1b18fe4"}
Oct 30 22:15:01.563716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.563690    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerStarted","Data":"85793da1f474726fd6d6381e8b0e7e45f8cf661b93f82aae28ebcb104beef6c6"}
Oct 30 22:15:01.563889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.563744    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerStarted","Data":"5be1110283511b1844fef4fcbe70c10ff1efc8f8e607c5ed25842257055170f0"}
Oct 30 22:15:01.565271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.565249    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerStarted","Data":"0f4ac9eab40d052298570165a8d660efad16bc4e27b804887b318678e1a2bc6e"}
Oct 30 22:15:01.565358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.565289    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerStarted","Data":"9b6661e43cfd016b76a803e4be0b2c9c6f083dcdf29c3d195a2a82afa8e991c7"}
Oct 30 22:15:01.674721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.674664    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" podStartSLOduration=1.674651775 podStartE2EDuration="1.674651775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.671556381 +0000 UTC m=+1954.760294184" watchObservedRunningTime="2026-10-30 22:15:01.674651775 +0000 UTC m=+1954.763389577"
Oct 30 22:15:01.789585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.789536    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" podStartSLOduration=1.789523188 podStartE2EDuration="1.789523188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.787370395 +0000 UTC m=+1954.876108191" watchObservedRunningTime="2026-10-30 22:15:01.789523188 +0000 UTC m=+1954.878260990"
Oct 30 22:15:01.903933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:01.903887    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" podStartSLOduration=1.9038737669999999 podStartE2EDuration="1.903873767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.901928335 +0000 UTC m=+1954.990666139" watchObservedRunningTime="2026-10-30 22:15:01.903873767 +0000 UTC m=+1954.992611569"
Oct 30 22:15:02.019104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.019045    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" podStartSLOduration=2.019028343 podStartE2EDuration="2.019028343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.016007953 +0000 UTC m=+1955.104745757" watchObservedRunningTime="2026-10-30 22:15:02.019028343 +0000 UTC m=+1955.107766148"
Oct 30 22:15:02.573069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.572983    4410 generic.go:356] "Generic (PLEG): container finished" podID="9f143630-c95d-4376-8fde-5ab244b722c3" containerID="97e39fa4007a3b80d382442e3c3adb05639e4941544be4b88180848634d20916" exitCode=60
Oct 30 22:15:02.573211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.573076    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerDied","Data":"97e39fa4007a3b80d382442e3c3adb05639e4941544be4b88180848634d20916"}
Oct 30 22:15:02.573346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.573323    4410 scope.go:122] "RemoveContainer" containerID="97e39fa4007a3b80d382442e3c3adb05639e4941544be4b88180848634d20916"
Oct 30 22:15:02.574833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.574810    4410 generic.go:356] "Generic (PLEG): container finished" podID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerID="c71733118100b9777d02b8b29f8826f859c93cbb5c1f9d162ca742a0d7c16a49" exitCode=60
Oct 30 22:15:02.574944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.574839    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerDied","Data":"c71733118100b9777d02b8b29f8826f859c93cbb5c1f9d162ca742a0d7c16a49"}
Oct 30 22:15:02.575115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.575099    4410 scope.go:122] "RemoveContainer" containerID="c71733118100b9777d02b8b29f8826f859c93cbb5c1f9d162ca742a0d7c16a49"
Oct 30 22:15:02.576623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.576597    4410 generic.go:356] "Generic (PLEG): container finished" podID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerID="387b5d339b4377d6e97658fc1b7e42e396eac8d2df55221c5216df7dd8c18706" exitCode=60
Oct 30 22:15:02.576711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.576634    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerDied","Data":"387b5d339b4377d6e97658fc1b7e42e396eac8d2df55221c5216df7dd8c18706"}
Oct 30 22:15:02.576888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.576870    4410 scope.go:122] "RemoveContainer" containerID="387b5d339b4377d6e97658fc1b7e42e396eac8d2df55221c5216df7dd8c18706"
Oct 30 22:15:02.578338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.578315    4410 generic.go:356] "Generic (PLEG): container finished" podID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerID="bab65126f7bf1f84d90d1f4a7df8813c5b01e54fb9bce77bf11fa29cb3b1310c" exitCode=60
Oct 30 22:15:02.578418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.578360    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerDied","Data":"bab65126f7bf1f84d90d1f4a7df8813c5b01e54fb9bce77bf11fa29cb3b1310c"}
Oct 30 22:15:02.578605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.578588    4410 scope.go:122] "RemoveContainer" containerID="bab65126f7bf1f84d90d1f4a7df8813c5b01e54fb9bce77bf11fa29cb3b1310c"
Oct 30 22:15:02.579945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.579926    4410 generic.go:356] "Generic (PLEG): container finished" podID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerID="85793da1f474726fd6d6381e8b0e7e45f8cf661b93f82aae28ebcb104beef6c6" exitCode=60
Oct 30 22:15:02.580055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.579998    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerDied","Data":"85793da1f474726fd6d6381e8b0e7e45f8cf661b93f82aae28ebcb104beef6c6"}
Oct 30 22:15:02.580193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.580178    4410 scope.go:122] "RemoveContainer" containerID="85793da1f474726fd6d6381e8b0e7e45f8cf661b93f82aae28ebcb104beef6c6"
Oct 30 22:15:02.581354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.581333    4410 generic.go:356] "Generic (PLEG): container finished" podID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerID="0f4ac9eab40d052298570165a8d660efad16bc4e27b804887b318678e1a2bc6e" exitCode=60
Oct 30 22:15:02.581443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.581352    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerDied","Data":"0f4ac9eab40d052298570165a8d660efad16bc4e27b804887b318678e1a2bc6e"}
Oct 30 22:15:02.581568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.581555    4410 scope.go:122] "RemoveContainer" containerID="0f4ac9eab40d052298570165a8d660efad16bc4e27b804887b318678e1a2bc6e"
Oct 30 22:15:02.900705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:02.900676    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.588807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.588761    4410 generic.go:356] "Generic (PLEG): container finished" podID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerID="652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b" exitCode=60
Oct 30 22:15:03.588904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.588814    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerDied","Data":"652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b"}
Oct 30 22:15:03.588904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.588873    4410 scope.go:122] "RemoveContainer" containerID="387b5d339b4377d6e97658fc1b7e42e396eac8d2df55221c5216df7dd8c18706"
Oct 30 22:15:03.589115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.589100    4410 scope.go:122] "RemoveContainer" containerID="652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b"
Oct 30 22:15:03.589431 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:03.589399    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm_odh-ai-gateway-infra(b2e22984-492a-4fe3-a47a-2fe15f5e06e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1"
Oct 30 22:15:03.591414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.591058    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerStarted","Data":"9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"}
Oct 30 22:15:03.592893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.592873    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerStarted","Data":"31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"}
Oct 30 22:15:03.594839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.594812    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerStarted","Data":"8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"}
Oct 30 22:15:03.596740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.596722    4410 generic.go:356] "Generic (PLEG): container finished" podID="9f143630-c95d-4376-8fde-5ab244b722c3" containerID="927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8" exitCode=60
Oct 30 22:15:03.596839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.596803    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerDied","Data":"927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8"}
Oct 30 22:15:03.597028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.597013    4410 scope.go:122] "RemoveContainer" containerID="927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8"
Oct 30 22:15:03.597246 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:03.597224    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx_odh-ai-gateway-infra(9f143630-c95d-4376-8fde-5ab244b722c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" podUID="9f143630-c95d-4376-8fde-5ab244b722c3"
Oct 30 22:15:03.598486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.598469    4410 generic.go:356] "Generic (PLEG): container finished" podID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerID="5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379" exitCode=60
Oct 30 22:15:03.598561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.598498    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerDied","Data":"5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379"}
Oct 30 22:15:03.598701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.598686    4410 scope.go:122] "RemoveContainer" containerID="5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379"
Oct 30 22:15:03.598902 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:03.598885    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9_odh-ai-gateway-infra(d8e81557-79fc-44b4-85df-69b4e259e9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea"
Oct 30 22:15:03.936271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.936249    4410 scope.go:122] "RemoveContainer" containerID="97e39fa4007a3b80d382442e3c3adb05639e4941544be4b88180848634d20916"
Oct 30 22:15:03.948792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:03.948747    4410 scope.go:122] "RemoveContainer" containerID="c71733118100b9777d02b8b29f8826f859c93cbb5c1f9d162ca742a0d7c16a49"
Oct 30 22:15:04.279104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.279048    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" podStartSLOduration=4.279034819 podStartE2EDuration="4.279034819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.277357999 +0000 UTC m=+1957.366095801" watchObservedRunningTime="2026-10-30 22:15:04.279034819 +0000 UTC m=+1957.367772621"
Oct 30 22:15:04.606819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.606708    4410 generic.go:356] "Generic (PLEG): container finished" podID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerID="31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a" exitCode=60
Oct 30 22:15:04.606819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.606789    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerDied","Data":"31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"}
Oct 30 22:15:04.607067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.606872    4410 scope.go:122] "RemoveContainer" containerID="85793da1f474726fd6d6381e8b0e7e45f8cf661b93f82aae28ebcb104beef6c6"
Oct 30 22:15:04.607156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.607135    4410 scope.go:122] "RemoveContainer" containerID="31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"
Oct 30 22:15:04.607453 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.607426    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j_odh-ai-gateway-infra(ec2d087e-1c88-4a16-874f-01c77c4a46ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac"
Oct 30 22:15:04.608646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.608624    4410 generic.go:356] "Generic (PLEG): container finished" podID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerID="8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736" exitCode=60
Oct 30 22:15:04.608730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.608708    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerDied","Data":"8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"}
Oct 30 22:15:04.608944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.608930    4410 scope.go:122] "RemoveContainer" containerID="8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"
Oct 30 22:15:04.609206 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.609185    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww_odh-ai-gateway-infra(58aabec7-5d5f-40a4-8b04-fd488eec2bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc"
Oct 30 22:15:04.610464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.610445    4410 scope.go:122] "RemoveContainer" containerID="927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8"
Oct 30 22:15:04.610701 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.610684    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx_odh-ai-gateway-infra(9f143630-c95d-4376-8fde-5ab244b722c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" podUID="9f143630-c95d-4376-8fde-5ab244b722c3"
Oct 30 22:15:04.611877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.611857    4410 scope.go:122] "RemoveContainer" containerID="5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379"
Oct 30 22:15:04.612137 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.612119    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9_odh-ai-gateway-infra(d8e81557-79fc-44b4-85df-69b4e259e9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea"
Oct 30 22:15:04.613364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.613350    4410 scope.go:122] "RemoveContainer" containerID="652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b"
Oct 30 22:15:04.613608 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.613584    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm_odh-ai-gateway-infra(b2e22984-492a-4fe3-a47a-2fe15f5e06e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1"
Oct 30 22:15:04.616963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.616940    4410 generic.go:356] "Generic (PLEG): container finished" podID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerID="9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3" exitCode=60
Oct 30 22:15:04.617052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.616982    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerDied","Data":"9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"}
Oct 30 22:15:04.617213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.617197    4410 scope.go:122] "RemoveContainer" containerID="9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"
Oct 30 22:15:04.617418 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:04.617398    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r_odh-ai-gateway-infra(6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c"
Oct 30 22:15:04.620483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.620466    4410 scope.go:122] "RemoveContainer" containerID="0f4ac9eab40d052298570165a8d660efad16bc4e27b804887b318678e1a2bc6e"
Oct 30 22:15:04.632632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:04.632604    4410 scope.go:122] "RemoveContainer" containerID="bab65126f7bf1f84d90d1f4a7df8813c5b01e54fb9bce77bf11fa29cb3b1310c"
Oct 30 22:15:05.625366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:05.625335    4410 scope.go:122] "RemoveContainer" containerID="9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"
Oct 30 22:15:05.625820 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:05.625654    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r_odh-ai-gateway-infra(6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c"
Oct 30 22:15:05.627157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:05.627135    4410 scope.go:122] "RemoveContainer" containerID="31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"
Oct 30 22:15:05.627413 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:05.627388    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j_odh-ai-gateway-infra(ec2d087e-1c88-4a16-874f-01c77c4a46ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac"
Oct 30 22:15:05.628594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:05.628577    4410 scope.go:122] "RemoveContainer" containerID="8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"
Oct 30 22:15:05.628800 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:05.628759    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww_odh-ai-gateway-infra(58aabec7-5d5f-40a4-8b04-fd488eec2bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc"
Oct 30 22:15:07.860177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:07.860143    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:08.503437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:08.503403    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:09.158426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:09.158385    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:09.319783 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:09.319742    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:09.475939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:09.475858    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:10.032611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:10.032573    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:10.285056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:10.284947    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.059101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:11.059063    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"]
Oct 30 22:15:11.106379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:11.106341    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a77a61c-7gcvl"]
Oct 30 22:15:12.850374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:12.850330    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.021481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:13.021449    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa6b067-1c17-46a8-918f-ab71b781647a" path="/var/lib/kubelet/pods/1fa6b067-1c17-46a8-918f-ab71b781647a/volumes"
Oct 30 22:15:15.407460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:15.407424    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:15.584518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:15.584476    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:15.755921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:15.755810    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.012802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.012714    4410 scope.go:122] "RemoveContainer" containerID="927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8"
Oct 30 22:15:16.012978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.012866    4410 scope.go:122] "RemoveContainer" containerID="8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"
Oct 30 22:15:16.013071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.012995    4410 scope.go:122] "RemoveContainer" containerID="31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"
Oct 30 22:15:16.398067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.398044    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"]
Oct 30 22:15:16.401851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.401832    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.458531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.458504    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:16.462836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.462812    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"]
Oct 30 22:15:16.514766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.514739    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfts8\" (UniqueName: \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.514922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.514769    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.514998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.514982    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.616152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.616080    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.616282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.616180    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfts8\" (UniqueName: \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.616282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.616199    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.616787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.616751    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.618476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.618450    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.674157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.674124    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfts8\" (UniqueName: \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\") pod \"maas-api-revoke-sub-09489e22-98784\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:16.703761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.703721    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerStarted","Data":"0d9a26b481a18290475ae4ba2ee89b8759510639767d28bacb463e092712c7f3"}
Oct 30 22:15:16.705639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.705616    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerStarted","Data":"1673ee8c4fe99b98f66941f2758ae5a96f4fd9b8f02287df51c1156a34e67bd7"}
Oct 30 22:15:16.707390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.707370    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerStarted","Data":"08ab643b940f2a571839ba982a7c192e115243d973c0421eaa4d4b3e4e4c22db"}
Oct 30 22:15:16.714748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:16.714727    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:17.013032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.013008    4410 scope.go:122] "RemoveContainer" containerID="9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"
Oct 30 22:15:17.087769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.087745    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"]
Oct 30 22:15:17.134767 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:17.134731    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa82f573_3bc0_4f8d_9b3d_699e0f6e9c0b.slice/crio-719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5 WatchSource:0}: Error finding container 719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5: Status 404 returned error can't find the container with id 719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5
Oct 30 22:15:17.715654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.715619    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerStarted","Data":"f8267d1158be020ab17e06754502c8cbe13f659377f774cf89b97f4fdcd8c89d"}
Oct 30 22:15:17.717653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.717630    4410 generic.go:356] "Generic (PLEG): container finished" podID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerID="1673ee8c4fe99b98f66941f2758ae5a96f4fd9b8f02287df51c1156a34e67bd7" exitCode=60
Oct 30 22:15:17.717748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.717701    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerDied","Data":"1673ee8c4fe99b98f66941f2758ae5a96f4fd9b8f02287df51c1156a34e67bd7"}
Oct 30 22:15:17.717826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.717752    4410 scope.go:122] "RemoveContainer" containerID="31ce6d2d6a6a7f142f671ddfae747ddb452570b2dc2130227e29678184df426a"
Oct 30 22:15:17.718094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.718076    4410 scope.go:122] "RemoveContainer" containerID="1673ee8c4fe99b98f66941f2758ae5a96f4fd9b8f02287df51c1156a34e67bd7"
Oct 30 22:15:17.718380 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:17.718353    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j_odh-ai-gateway-infra(ec2d087e-1c88-4a16-874f-01c77c4a46ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac"
Oct 30 22:15:17.719705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.719688    4410 generic.go:356] "Generic (PLEG): container finished" podID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerID="08ab643b940f2a571839ba982a7c192e115243d973c0421eaa4d4b3e4e4c22db" exitCode=60
Oct 30 22:15:17.719795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.719752    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerDied","Data":"08ab643b940f2a571839ba982a7c192e115243d973c0421eaa4d4b3e4e4c22db"}
Oct 30 22:15:17.720084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.720066    4410 scope.go:122] "RemoveContainer" containerID="08ab643b940f2a571839ba982a7c192e115243d973c0421eaa4d4b3e4e4c22db"
Oct 30 22:15:17.720309 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:17.720290    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww_odh-ai-gateway-infra(58aabec7-5d5f-40a4-8b04-fd488eec2bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc"
Oct 30 22:15:17.721402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.721381    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784" event={"ID":"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b","Type":"ContainerStarted","Data":"12dad544a364964b2b3d15a8840c01e713eba2b52166c6c09a1b6c9ab0ad2dac"}
Oct 30 22:15:17.721527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.721418    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784" event={"ID":"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b","Type":"ContainerStarted","Data":"719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5"}
Oct 30 22:15:17.723296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.723279    4410 generic.go:356] "Generic (PLEG): container finished" podID="9f143630-c95d-4376-8fde-5ab244b722c3" containerID="0d9a26b481a18290475ae4ba2ee89b8759510639767d28bacb463e092712c7f3" exitCode=60
Oct 30 22:15:17.723378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.723342    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerDied","Data":"0d9a26b481a18290475ae4ba2ee89b8759510639767d28bacb463e092712c7f3"}
Oct 30 22:15:17.723591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.723578    4410 scope.go:122] "RemoveContainer" containerID="0d9a26b481a18290475ae4ba2ee89b8759510639767d28bacb463e092712c7f3"
Oct 30 22:15:17.723786 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:17.723760    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx_odh-ai-gateway-infra(9f143630-c95d-4376-8fde-5ab244b722c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" podUID="9f143630-c95d-4376-8fde-5ab244b722c3"
Oct 30 22:15:17.730345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.730326    4410 scope.go:122] "RemoveContainer" containerID="8c16ab873a2b458f22f99ea13b4bd3da5dc089b21fec384e48ebe8012783b736"
Oct 30 22:15:17.743492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.743475    4410 scope.go:122] "RemoveContainer" containerID="927b99f86017c48af598b5e41e492ba9275262d86f3b905de06bff62793136e8"
Oct 30 22:15:17.827902 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.827872    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:17.943634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:17.943604    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:18.012472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.012390    4410 scope.go:122] "RemoveContainer" containerID="652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b"
Oct 30 22:15:18.012891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.012870    4410 scope.go:122] "RemoveContainer" containerID="5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379"
Oct 30 22:15:18.020080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.020055    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:18.058530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.058498    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:18.066828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.066739    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784" podStartSLOduration=2.066704309 podStartE2EDuration="2.066704309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:18.063897676 +0000 UTC m=+1971.152635480" watchObservedRunningTime="2026-10-30 22:15:18.066704309 +0000 UTC m=+1971.155442112"
Oct 30 22:15:18.662728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.662697    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:18.732795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.732744    4410 generic.go:356] "Generic (PLEG): container finished" podID="aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" containerID="12dad544a364964b2b3d15a8840c01e713eba2b52166c6c09a1b6c9ab0ad2dac" exitCode=0
Oct 30 22:15:18.734561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.732878    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784" event={"ID":"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b","Type":"ContainerDied","Data":"12dad544a364964b2b3d15a8840c01e713eba2b52166c6c09a1b6c9ab0ad2dac"}
Oct 30 22:15:18.738383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.738322    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerStarted","Data":"fc3fba01cd861b5d720d9d64f36c7f064750dd6ca5f7474cce7a6b1f884c4c04"}
Oct 30 22:15:18.742026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.741992    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerStarted","Data":"e80369aaff015fb33f5188f6b6988c7b1b5d644f4049876dd6dd62a242211c3d"}
Oct 30 22:15:18.747820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.744817    4410 generic.go:356] "Generic (PLEG): container finished" podID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerID="f8267d1158be020ab17e06754502c8cbe13f659377f774cf89b97f4fdcd8c89d" exitCode=60
Oct 30 22:15:18.747820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.744929    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerDied","Data":"f8267d1158be020ab17e06754502c8cbe13f659377f774cf89b97f4fdcd8c89d"}
Oct 30 22:15:18.747820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.744978    4410 scope.go:122] "RemoveContainer" containerID="9dbd8bc404c366e666fe13adb1791c91e8b98b3f1891177ec5d84aaeaddd26a3"
Oct 30 22:15:18.747820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.745326    4410 scope.go:122] "RemoveContainer" containerID="f8267d1158be020ab17e06754502c8cbe13f659377f774cf89b97f4fdcd8c89d"
Oct 30 22:15:18.749591 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:18.748327    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r_odh-ai-gateway-infra(6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c"
Oct 30 22:15:18.859788 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.847381    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:18.893815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:18.892079    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:19.012616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.012586    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:19.024411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.024383    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:19.061468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.061444    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185323    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\" (UniqueName: \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\") pod \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") "
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185381    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\") pod \"9f143630-c95d-4376-8fde-5ab244b722c3\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") "
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185435    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\" (UniqueName: \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\") pod \"9f143630-c95d-4376-8fde-5ab244b722c3\" (UID: \"9f143630-c95d-4376-8fde-5ab244b722c3\") "
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185511    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\" (UniqueName: \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\") pod \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") "
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185539    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\") pod \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\" (UID: \"ec2d087e-1c88-4a16-874f-01c77c4a46ac\") "
Oct 30 22:15:19.196888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.185648    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\") pod \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\" (UID: \"58aabec7-5d5f-40a4-8b04-fd488eec2bcc\") "
Oct 30 22:15:19.198082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.197749    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca" pod "9f143630-c95d-4376-8fde-5ab244b722c3" (UID: "9f143630-c95d-4376-8fde-5ab244b722c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.201467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.201437    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca" pod "58aabec7-5d5f-40a4-8b04-fd488eec2bcc" (UID: "58aabec7-5d5f-40a4-8b04-fd488eec2bcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.203017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.202980    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca" pod "ec2d087e-1c88-4a16-874f-01c77c4a46ac" (UID: "ec2d087e-1c88-4a16-874f-01c77c4a46ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.203344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.203295    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff" pod "ec2d087e-1c88-4a16-874f-01c77c4a46ac" (UID: "ec2d087e-1c88-4a16-874f-01c77c4a46ac"). InnerVolumeSpecName "kube-api-access-ntzff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.206583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.206539    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm" pod "9f143630-c95d-4376-8fde-5ab244b722c3" (UID: "9f143630-c95d-4376-8fde-5ab244b722c3"). InnerVolumeSpecName "kube-api-access-s6xdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.210056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.210026    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz" pod "58aabec7-5d5f-40a4-8b04-fd488eec2bcc" (UID: "58aabec7-5d5f-40a4-8b04-fd488eec2bcc"). InnerVolumeSpecName "kube-api-access-l7wrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.286910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286881    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xdm\" (UniqueName: \"kubernetes.io/projected/9f143630-c95d-4376-8fde-5ab244b722c3-kube-api-access-s6xdm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.286910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286908    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wrz\" (UniqueName: \"kubernetes.io/projected/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-kube-api-access-l7wrz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.287125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286917    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d087e-1c88-4a16-874f-01c77c4a46ac-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.287125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286927    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aabec7-5d5f-40a4-8b04-fd488eec2bcc-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.287125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286936    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntzff\" (UniqueName: \"kubernetes.io/projected/ec2d087e-1c88-4a16-874f-01c77c4a46ac-kube-api-access-ntzff\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.287125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.286944    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f143630-c95d-4376-8fde-5ab244b722c3-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.352377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.352324    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" podStartSLOduration=19.352310329 podStartE2EDuration="19.352310329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:19.350881496 +0000 UTC m=+1972.439619312" watchObservedRunningTime="2026-10-30 22:15:19.352310329 +0000 UTC m=+1972.441048132"
Oct 30 22:15:19.603377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603297    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"]
Oct 30 22:15:19.603865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603848    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.603921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603867    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.603921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603880    4410 cpu_manager.go:414] "containerMap: removing container" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.603921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603885    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.603921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603902    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.603921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603907    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603932    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603941    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603960    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603965    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603979    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603983    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603990    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.603994    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604004    4410 cpu_manager.go:414] "containerMap: removing container" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.604071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604009    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604085    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604095    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604101    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604107    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604114    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604120    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604126    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604132    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" containerName="cleanup"
Oct 30 22:15:19.604357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.604138    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" containerName="cleanup"
Oct 30 22:15:19.609121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.609089    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.665437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.665404    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"]
Oct 30 22:15:19.692030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.691470    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxpb\" (UniqueName: \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.692030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.691691    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.692030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.691739    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.776582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.776551    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"
Oct 30 22:15:19.776582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.776552    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j" event={"ID":"ec2d087e-1c88-4a16-874f-01c77c4a46ac","Type":"ContainerDied","Data":"5be1110283511b1844fef4fcbe70c10ff1efc8f8e607c5ed25842257055170f0"}
Oct 30 22:15:19.777045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.776628    4410 scope.go:122] "RemoveContainer" containerID="1673ee8c4fe99b98f66941f2758ae5a96f4fd9b8f02287df51c1156a34e67bd7"
Oct 30 22:15:19.778065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.778048    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"
Oct 30 22:15:19.778261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.778049    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww" event={"ID":"58aabec7-5d5f-40a4-8b04-fd488eec2bcc","Type":"ContainerDied","Data":"9b6661e43cfd016b76a803e4be0b2c9c6f083dcdf29c3d195a2a82afa8e991c7"}
Oct 30 22:15:19.779807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.779763    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx" event={"ID":"9f143630-c95d-4376-8fde-5ab244b722c3","Type":"ContainerDied","Data":"d2fc09babcbf805c3f999ecb0130c77f58324eb8b2e119b3c0a542b5bd5444fd"}
Oct 30 22:15:19.779913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.779807    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"
Oct 30 22:15:19.789915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.789895    4410 generic.go:356] "Generic (PLEG): container finished" podID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerID="fc3fba01cd861b5d720d9d64f36c7f064750dd6ca5f7474cce7a6b1f884c4c04" exitCode=60
Oct 30 22:15:19.790831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.789993    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerDied","Data":"fc3fba01cd861b5d720d9d64f36c7f064750dd6ca5f7474cce7a6b1f884c4c04"}
Oct 30 22:15:19.790831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.790401    4410 scope.go:122] "RemoveContainer" containerID="fc3fba01cd861b5d720d9d64f36c7f064750dd6ca5f7474cce7a6b1f884c4c04"
Oct 30 22:15:19.791045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.790954    4410 scope.go:122] "RemoveContainer" containerID="08ab643b940f2a571839ba982a7c192e115243d973c0421eaa4d4b3e4e4c22db"
Oct 30 22:15:19.791303 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:19.791275    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9_odh-ai-gateway-infra(d8e81557-79fc-44b4-85df-69b4e259e9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea"
Oct 30 22:15:19.792523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.792490    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.792610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.792591    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxpb\" (UniqueName: \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.792804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.792768    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.794903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.793497    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.795627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.795591    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.798894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.798869    4410 generic.go:356] "Generic (PLEG): container finished" podID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerID="e80369aaff015fb33f5188f6b6988c7b1b5d644f4049876dd6dd62a242211c3d" exitCode=60
Oct 30 22:15:19.798964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.798939    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerDied","Data":"e80369aaff015fb33f5188f6b6988c7b1b5d644f4049876dd6dd62a242211c3d"}
Oct 30 22:15:19.799411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.799394    4410 scope.go:122] "RemoveContainer" containerID="e80369aaff015fb33f5188f6b6988c7b1b5d644f4049876dd6dd62a242211c3d"
Oct 30 22:15:19.799678 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:19.799649    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm_odh-ai-gateway-infra(b2e22984-492a-4fe3-a47a-2fe15f5e06e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1"
Oct 30 22:15:19.820452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.820427    4410 scope.go:122] "RemoveContainer" containerID="0d9a26b481a18290475ae4ba2ee89b8759510639767d28bacb463e092712c7f3"
Oct 30 22:15:19.832460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.832428    4410 scope.go:122] "RemoveContainer" containerID="5e910df5c35879256c1d7386140caab34b995f452c7162dc7320a447cd048379"
Oct 30 22:15:19.850862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.850836    4410 scope.go:122] "RemoveContainer" containerID="652b83745b633bae3a120a7c0f48ca00939dee603e551b2c8e7891307c7ff07b"
Oct 30 22:15:19.855442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.855383    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxpb\" (UniqueName: \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\") pod \"maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.927739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.927702    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:19.953984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.953963    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:19.986255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:19.986230    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:20.031922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.031892    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:20.034282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.034258    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889975-xhg5j"]
Oct 30 22:15:20.096195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096168    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\") pod \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") "
Oct 30 22:15:20.096337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096220    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\" (UniqueName: \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\") pod \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") "
Oct 30 22:15:20.096337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096247    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\") pod \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\" (UID: \"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c\") "
Oct 30 22:15:20.096469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096338    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\" (UniqueName: \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\") pod \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") "
Oct 30 22:15:20.096469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096370    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\") pod \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\" (UID: \"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b\") "
Oct 30 22:15:20.096668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096637    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca" pod "6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" (UID: "6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.096879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096838    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca" pod "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" (UID: "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.096879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.096857    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.098513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.098489    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca" pod "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" (UID: "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:20.098724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.098698    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc" pod "6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" (UID: "6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c"). InnerVolumeSpecName "kube-api-access-g4mnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.098724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.098698    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8" pod "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" (UID: "aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b"). InnerVolumeSpecName "kube-api-access-bfts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.197808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.197749    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mnc\" (UniqueName: \"kubernetes.io/projected/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c-kube-api-access-g4mnc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.197808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.197796    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfts8\" (UniqueName: \"kubernetes.io/projected/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-kube-api-access-bfts8\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.197808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.197807    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.198027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.197817    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.253739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.253707    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:20.257795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.257755    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29889975-wd4ww"]
Oct 30 22:15:20.310174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.310153    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"]
Oct 30 22:15:20.312392 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:20.312355    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8592177f_6efd_4dd8_a7ba_dc58e2147ec4.slice/crio-66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09 WatchSource:0}: Error finding container 66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09: Status 404 returned error can't find the container with id 66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09
Oct 30 22:15:20.361254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.361198    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:20.480904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.480871    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:20.491646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.491619    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-97ca39-29889975-r29vx"]
Oct 30 22:15:20.530798 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:20.530736    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:15:20.587839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.587806    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:20.808657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.808633    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"
Oct 30 22:15:20.809112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.808660    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784" event={"ID":"aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b","Type":"ContainerDied","Data":"719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5"}
Oct 30 22:15:20.809112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.808710    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719e787848e9d58d5b75840af1751621ca116e88206661e43586b034852acae5"
Oct 30 22:15:20.811257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.811230    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5" event={"ID":"8592177f-6efd-4dd8-a7ba-dc58e2147ec4","Type":"ContainerStarted","Data":"c1b05809ca96f07030816cdd3b3836b1043b9c53b62f7829fe7e296869d6d606"}
Oct 30 22:15:20.811375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.811278    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5" event={"ID":"8592177f-6efd-4dd8-a7ba-dc58e2147ec4","Type":"ContainerStarted","Data":"66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09"}
Oct 30 22:15:20.820436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.820399    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r" event={"ID":"6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c","Type":"ContainerDied","Data":"95cc8bacd1526cbf3cdaca4923a74ed6393b46557c1cbf95c78be14bf1b18fe4"}
Oct 30 22:15:20.820436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.820432    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"
Oct 30 22:15:20.820601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.820455    4410 scope.go:122] "RemoveContainer" containerID="f8267d1158be020ab17e06754502c8cbe13f659377f774cf89b97f4fdcd8c89d"
Oct 30 22:15:20.971741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:20.971705    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:21.005862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.005834    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:21.020927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.020895    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" path="/var/lib/kubelet/pods/58aabec7-5d5f-40a4-8b04-fd488eec2bcc/volumes"
Oct 30 22:15:21.021467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.021451    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f143630-c95d-4376-8fde-5ab244b722c3" path="/var/lib/kubelet/pods/9f143630-c95d-4376-8fde-5ab244b722c3/volumes"
Oct 30 22:15:21.021974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.021959    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d087e-1c88-4a16-874f-01c77c4a46ac" path="/var/lib/kubelet/pods/ec2d087e-1c88-4a16-874f-01c77c4a46ac/volumes"
Oct 30 22:15:21.051808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.051767    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:21.054382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.054357    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29889975-zbs6r"]
Oct 30 22:15:21.108287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108263    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\") pod \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") "
Oct 30 22:15:21.108431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108371    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\" (UniqueName: \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\") pod \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\" (UID: \"b2e22984-492a-4fe3-a47a-2fe15f5e06e1\") "
Oct 30 22:15:21.108502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108487    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\") pod \"d8e81557-79fc-44b4-85df-69b4e259e9ea\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") "
Oct 30 22:15:21.108602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108586    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\" (UniqueName: \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\") pod \"d8e81557-79fc-44b4-85df-69b4e259e9ea\" (UID: \"d8e81557-79fc-44b4-85df-69b4e259e9ea\") "
Oct 30 22:15:21.108659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108637    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca" pod "b2e22984-492a-4fe3-a47a-2fe15f5e06e1" (UID: "b2e22984-492a-4fe3-a47a-2fe15f5e06e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.108918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.108893    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca" pod "d8e81557-79fc-44b4-85df-69b4e259e9ea" (UID: "d8e81557-79fc-44b4-85df-69b4e259e9ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.110360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.109530    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e81557-79fc-44b4-85df-69b4e259e9ea-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.110360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.109581    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.119019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.118925    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn" pod "b2e22984-492a-4fe3-a47a-2fe15f5e06e1" (UID: "b2e22984-492a-4fe3-a47a-2fe15f5e06e1"). InnerVolumeSpecName "kube-api-access-ntkqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.119182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.119161    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6" pod "d8e81557-79fc-44b4-85df-69b4e259e9ea" (UID: "d8e81557-79fc-44b4-85df-69b4e259e9ea"). InnerVolumeSpecName "kube-api-access-mp4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.210427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.210366    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4g6\" (UniqueName: \"kubernetes.io/projected/d8e81557-79fc-44b4-85df-69b4e259e9ea-kube-api-access-mp4g6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.210427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.210390    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkqn\" (UniqueName: \"kubernetes.io/projected/b2e22984-492a-4fe3-a47a-2fe15f5e06e1-kube-api-access-ntkqn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.226045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.225995    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5" podStartSLOduration=2.225711713 podStartE2EDuration="2.225711713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.224712929 +0000 UTC m=+1974.313450742" watchObservedRunningTime="2026-10-30 22:15:21.225711713 +0000 UTC m=+1974.314449517"
Oct 30 22:15:21.828111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.828077    4410 generic.go:356] "Generic (PLEG): container finished" podID="8592177f-6efd-4dd8-a7ba-dc58e2147ec4" containerID="c1b05809ca96f07030816cdd3b3836b1043b9c53b62f7829fe7e296869d6d606" exitCode=0
Oct 30 22:15:21.828570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.828165    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5" event={"ID":"8592177f-6efd-4dd8-a7ba-dc58e2147ec4","Type":"ContainerDied","Data":"c1b05809ca96f07030816cdd3b3836b1043b9c53b62f7829fe7e296869d6d606"}
Oct 30 22:15:21.829562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.829541    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"
Oct 30 22:15:21.829562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.829554    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9" event={"ID":"d8e81557-79fc-44b4-85df-69b4e259e9ea","Type":"ContainerDied","Data":"5fa3ffb06efeec7369f83361c7116e66683d47a15b917eaa65e797689ee6fccb"}
Oct 30 22:15:21.829721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.829611    4410 scope.go:122] "RemoveContainer" containerID="fc3fba01cd861b5d720d9d64f36c7f064750dd6ca5f7474cce7a6b1f884c4c04"
Oct 30 22:15:21.831041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.831016    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"
Oct 30 22:15:21.831131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.831019    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm" event={"ID":"b2e22984-492a-4fe3-a47a-2fe15f5e06e1","Type":"ContainerDied","Data":"be2e4a8d520e1d770d4fa699dc6d3df5f2e2db5730b1aaacea9ed9a12342033a"}
Oct 30 22:15:21.840620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:21.840598    4410 scope.go:122] "RemoveContainer" containerID="e80369aaff015fb33f5188f6b6988c7b1b5d644f4049876dd6dd62a242211c3d"
Oct 30 22:15:22.076496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:22.076463    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:22.082999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:22.082973    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-859078-29889975-cxnb9"]
Oct 30 22:15:22.209989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:22.209955    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:22.231869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:22.231839    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29889975-7rnwm"]
Oct 30 22:15:22.980873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:22.980851    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:23.020605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.020578    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" path="/var/lib/kubelet/pods/6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c/volumes"
Oct 30 22:15:23.021009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.020994    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" path="/var/lib/kubelet/pods/b2e22984-492a-4fe3-a47a-2fe15f5e06e1/volumes"
Oct 30 22:15:23.021355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.021341    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" path="/var/lib/kubelet/pods/d8e81557-79fc-44b4-85df-69b4e259e9ea/volumes"
Oct 30 22:15:23.130953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.130876    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\" (UniqueName: \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\") pod \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") "
Oct 30 22:15:23.131111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.131002    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\") pod \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") "
Oct 30 22:15:23.131111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.131074    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\") pod \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\" (UID: \"8592177f-6efd-4dd8-a7ba-dc58e2147ec4\") "
Oct 30 22:15:23.131515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.131478    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca" pod "8592177f-6efd-4dd8-a7ba-dc58e2147ec4" (UID: "8592177f-6efd-4dd8-a7ba-dc58e2147ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:23.131640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.131565    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.132980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.132958    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb" pod "8592177f-6efd-4dd8-a7ba-dc58e2147ec4" (UID: "8592177f-6efd-4dd8-a7ba-dc58e2147ec4"). InnerVolumeSpecName "kube-api-access-dqxpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:23.133812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.133794    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca" pod "8592177f-6efd-4dd8-a7ba-dc58e2147ec4" (UID: "8592177f-6efd-4dd8-a7ba-dc58e2147ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:23.232228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.232206    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxpb\" (UniqueName: \"kubernetes.io/projected/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-kube-api-access-dqxpb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.232228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.232229    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8592177f-6efd-4dd8-a7ba-dc58e2147ec4-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.591558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.591523    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:23.849214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.849083    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"
Oct 30 22:15:23.849214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.849118    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5" event={"ID":"8592177f-6efd-4dd8-a7ba-dc58e2147ec4","Type":"ContainerDied","Data":"66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09"}
Oct 30 22:15:23.849214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:23.849162    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66adea6664c6d565415396abe20214f56b62e168285cbf941f4c69b437c12c09"
Oct 30 22:15:24.799467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:24.799431    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:25.608325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:25.608282    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:26.570241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:26.570194    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.151712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.151675    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.450195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.450116    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.696741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.696668    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.887673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.887582    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.933715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.933681    4410 scope.go:122] "RemoveContainer" containerID="72636161062bb84c8c1c54b105e7b15c3bddbb552e049489672ffa8da884e9d7"
Oct 30 22:15:27.948269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:27.948186    4410 scope.go:122] "RemoveContainer" containerID="0f8dadafe5c4f3046273b8455fd74e1065dfc9df15a648716224449a20c050fe"
Oct 30 22:15:28.414558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.414520    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:15:28.415036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415018    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415037    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415047    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8592177f-6efd-4dd8-a7ba-dc58e2147ec4" containerName="revoke-keys"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415053    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8592177f-6efd-4dd8-a7ba-dc58e2147ec4" containerName="revoke-keys"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415070    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415076    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415082    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415086    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415095    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415101    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415119    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415123    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415133    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415137    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415147    4410 cpu_manager.go:414] "containerMap: removing container" podUID="aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" containerName="revoke-keys"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415151    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" containerName="revoke-keys"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415158    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415162    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415169    4410 cpu_manager.go:414] "containerMap: removing container" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415174    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aabec7-5d5f-40a4-8b04-fd488eec2bcc" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415235    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415242    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415249    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415254    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415262    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" containerName="revoke-keys"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415267    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415274    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415280    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:15:28.415363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.415286    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8592177f-6efd-4dd8-a7ba-dc58e2147ec4" containerName="revoke-keys"
Oct 30 22:15:28.420136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.420115    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.476799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.475829    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-577ea8-openshift-default-dockercfg-tjpzd\""
Oct 30 22:15:28.531148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.531116    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:15:28.631798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.630455    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633163    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633236    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633581    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633625    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633843    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.633890    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.634114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzx4\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.634175    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.635530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.634403    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.735970    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzx4\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736029    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736140    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736503    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736557    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.736612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736605    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736631    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736672    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736680    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736716    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.736972    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.737034    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.737175    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.737931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.737887    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.739055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.739009    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.739190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.739148    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.825804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.820336    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.825804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.821462    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzx4\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\") pod \"e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") " pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:28.829801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:28.826731    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:29.035876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.035000    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:29.574427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.574401    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:15:29.576300 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:15:29.576269    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a326d1_cf19_4060_97bc_be42cd1426ca.slice/crio-9f71895b6c6deb0c667a9effc604e308af2b52f7c1f7e75f8b1f042fdd855a48 WatchSource:0}: Error finding container 9f71895b6c6deb0c667a9effc604e308af2b52f7c1f7e75f8b1f042fdd855a48: Status 404 returned error can't find the container with id 9f71895b6c6deb0c667a9effc604e308af2b52f7c1f7e75f8b1f042fdd855a48
Oct 30 22:15:29.578649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.578615    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:29.578736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.578681    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:29.578736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.578711    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:29.923105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.923070    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerStarted","Data":"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"}
Oct 30 22:15:29.923565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:29.923134    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerStarted","Data":"9f71895b6c6deb0c667a9effc604e308af2b52f7c1f7e75f8b1f042fdd855a48"}
Oct 30 22:15:30.036342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.036310    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:30.038969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.038913    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podStartSLOduration=2.038897377 podStartE2EDuration="2.038897377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:30.038166954 +0000 UTC m=+1983.126904793" watchObservedRunningTime="2026-10-30 22:15:30.038897377 +0000 UTC m=+1983.127635184"
Oct 30 22:15:30.042564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.042542    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:30.576603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.576566    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:30.928698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.928668    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:30.929606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:30.929587    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:15:32.864290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:32.864257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:33.085485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:33.085450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:34.004272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:34.004229    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:35.799877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:35.799845    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:36.113565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:36.113482    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:36.295354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:36.295323    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:40.931298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:40.931267    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:44.557379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:44.557342    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.377903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:45.377861    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.595450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:45.595413    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:46.133216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:46.133183    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.251685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:48.251646    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.424609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:48.424575    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.278757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:49.278719    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.445388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:49.445339    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:50.521917 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:15:50.521881    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:15:51.620652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:51.620620    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:58.266153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:15:58.266116    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:02.196634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.196593    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:16:02.197261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197240    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:16:02.197362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197263    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e81557-79fc-44b4-85df-69b4e259e9ea" containerName="cleanup"
Oct 30 22:16:02.197362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197311    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:16:02.197362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197319    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:16:02.197524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197425    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e22984-492a-4fe3-a47a-2fe15f5e06e1" containerName="cleanup"
Oct 30 22:16:02.197524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.197441    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6502b9cd-a6c8-4fd8-9753-6c6bbc361b3c" containerName="cleanup"
Oct 30 22:16:02.200886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.200865    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.212701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.212675    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.212821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.212724    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.212821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.212810    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.212941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.212853    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.213005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.212941    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j5d\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.213057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.213016    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.213098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.213071    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.213147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.213096    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.213147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.213133    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.246344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.246320    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:16:02.249943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.249928    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.299851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.299826    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:16:02.313755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.313731    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j5d\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.313869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.313797    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.313869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.313824    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.313987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.313910    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.313987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.313961    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314033    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314075    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314108    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314177    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314235    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314247    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314266    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlw4\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314347    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314379    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314495    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314528    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314559    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314640    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314665    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314683    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.314895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314735    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.314895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.314816    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.315646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.315621    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.316864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.316844    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.317083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.317066    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.370707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.370674    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.370958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.370939    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j5d\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\") pod \"maas-default-gateway-openshift-default-694db55c87-j249g\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.403445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.403418    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:16:02.416415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416392    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416431    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416467    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416493    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw4\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416541    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416585    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416618    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416683    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.416963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.416806    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.417806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.417299    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.417806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.417541    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.418314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.418284    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.418404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.418372    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.418508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.418486    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.420441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.420410    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.420546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.420497    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.473340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.473280    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.473438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.473403    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlw4\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\") pod \"maas-default-gateway-openshift-default-694db55c87-lzkmk\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.514398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.514373    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:02.561814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.561765    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:02.940168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.940143    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:16:02.942016 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:16:02.941988    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d5355c_3025_47f3_8333_39439647415d.slice/crio-1c75c555b77eab8661a5ca9da52f965ad2071dcf226f3c87f0cee3dd13f7734e WatchSource:0}: Error finding container 1c75c555b77eab8661a5ca9da52f965ad2071dcf226f3c87f0cee3dd13f7734e: Status 404 returned error can't find the container with id 1c75c555b77eab8661a5ca9da52f965ad2071dcf226f3c87f0cee3dd13f7734e
Oct 30 22:16:02.944397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.944363    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:02.944495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.944426    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:02.944495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.944454    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:02.989420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.989398    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:16:02.990982 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:16:02.990951    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9fd184_8208_4667_b204_53d2e5e9581b.slice/crio-559759e80d079116f7f72c98be43a42aaad977daa7dcee68e1c5a0682a7ed33c WatchSource:0}: Error finding container 559759e80d079116f7f72c98be43a42aaad977daa7dcee68e1c5a0682a7ed33c: Status 404 returned error can't find the container with id 559759e80d079116f7f72c98be43a42aaad977daa7dcee68e1c5a0682a7ed33c
Oct 30 22:16:02.993754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.993717    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:02.993905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.993863    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:02.993969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:02.993945    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:03.120794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.120441    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" event={"ID":"52d5355c-3025-47f3-8333-39439647415d","Type":"ContainerStarted","Data":"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"}
Oct 30 22:16:03.120794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.120489    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" event={"ID":"52d5355c-3025-47f3-8333-39439647415d","Type":"ContainerStarted","Data":"1c75c555b77eab8661a5ca9da52f965ad2071dcf226f3c87f0cee3dd13f7734e"}
Oct 30 22:16:03.124361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.122382    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" event={"ID":"cc9fd184-8208-4667-b204-53d2e5e9581b","Type":"ContainerStarted","Data":"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"}
Oct 30 22:16:03.124361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.122435    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" event={"ID":"cc9fd184-8208-4667-b204-53d2e5e9581b","Type":"ContainerStarted","Data":"559759e80d079116f7f72c98be43a42aaad977daa7dcee68e1c5a0682a7ed33c"}
Oct 30 22:16:03.140646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.140620    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:03.240288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.240174    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podStartSLOduration=1.240155625 podStartE2EDuration="1.240155625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.236263433 +0000 UTC m=+2016.325001238" watchObservedRunningTime="2026-10-30 22:16:03.240155625 +0000 UTC m=+2016.328893429"
Oct 30 22:16:03.353544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.353484    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podStartSLOduration=1.353465646 podStartE2EDuration="1.353465646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.35281254 +0000 UTC m=+2016.441550343" watchObservedRunningTime="2026-10-30 22:16:03.353465646 +0000 UTC m=+2016.442203450"
Oct 30 22:16:03.514573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.514492    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:03.562823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:03.561877    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:04.516550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:04.516510    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.517063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:04.517027    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:04.574931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:04.574892    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.575239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:04.575194    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:05.515652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:05.515614    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.515849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:05.515664    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:05.562796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:05.562120    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.562796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:05.562181    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:06.515387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:06.515351    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.515572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:06.515407    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:06.564189 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:06.564147    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.564805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:06.564745    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:07.514877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:07.514844    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.515064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:07.514892    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:07.563214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:07.563172    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.564268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:07.564233    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:07.913759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:07.913707    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.188784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.188663    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.374641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.374590    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.514717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.514630    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.514922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.514715    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:08.566122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.566085    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.566675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:08.566641    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:09.515579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:09.515546    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.515818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:09.515599    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:09.562960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:09.562925    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.563098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:09.562979    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:10.516882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:10.516845    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j249g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.517518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:10.517478    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 22:16:10.562936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:10.562904    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lzkmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.563106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:10.562957    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 22:16:11.675884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.675844    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:11.676368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.676123    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:11.677045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.677024    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:16:11.694324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.694294    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:11.695622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.695602    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:11.696741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:11.696724    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:16:12.950851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:12.950817    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:18.001732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:18.001682    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.522433 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:16:20.522394    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:16:21.652936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:21.652902    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:22.110219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:22.110188    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.135652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:24.135619    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.707653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:24.707616    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.911225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:24.911192    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:25.198553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:25.198521    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:25.429560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:25.429515    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:25.829259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:25.829226    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:26.063427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:26.063389    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:26.231376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:26.231342    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:26.329691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:26.329663    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"]
Oct 30 22:16:26.378680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:26.378651    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97ca39-v2kj5"]
Oct 30 22:16:26.753522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:26.753484    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:27.020664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:27.020591    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8592177f-6efd-4dd8-a7ba-dc58e2147ec4" path="/var/lib/kubelet/pods/8592177f-6efd-4dd8-a7ba-dc58e2147ec4/volumes"
Oct 30 22:16:28.142927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:28.142886    4410 scope.go:122] "RemoveContainer" containerID="70a6e42c2349dd6a7f036cac1939136e01d825c84e971cafd46d677c862301b6"
Oct 30 22:16:28.154726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:28.154707    4410 scope.go:122] "RemoveContainer" containerID="9b4ee2fc6f191b21dd45f51d6bdd9c4a519c21a49ceead8c98f659554f8e7d16"
Oct 30 22:16:28.163190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:28.163171    4410 scope.go:122] "RemoveContainer" containerID="210861838a66c2edcb32376778ec9a1d8f46321fa765651ac3b8dc62f5181389"
Oct 30 22:16:30.990493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:30.990455    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:31.151701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:31.151662    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:31.325811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:31.325689    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.993408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:32.993368    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:33.207803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:33.207746    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:33.296394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:33.296313    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:16:33.296562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:33.296536    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy" containerID="cri-o://86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790" gracePeriod=30
Oct 30 22:16:33.372639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:33.372592    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.308907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:36.308863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:38.562612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.562554    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:16:38.738964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.738925    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.738996    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739029    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739097    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739127    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739162    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739212    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739331    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:38.739579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739389    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:38.739579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739474    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\") pod \"790f31ca-211c-4e39-9ab3-6158bbe697b6\" (UID: \"790f31ca-211c-4e39-9ab3-6158bbe697b6\") "
Oct 30 22:16:38.739579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.739548    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:38.743014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.742967    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "kube-api-access-bhr79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:38.743159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743076    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:38.743159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743090    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.743159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743135    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.743159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743151    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.743395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743167    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:38.743395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743253    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:38.743395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743272    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:38.743564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.743371    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs" pod "790f31ca-211c-4e39-9ab3-6158bbe697b6" (UID: "790f31ca-211c-4e39-9ab3-6158bbe697b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:38.844372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844293    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.844372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844328    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.844372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844342    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.844372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844356    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790f31ca-211c-4e39-9ab3-6158bbe697b6-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.844372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844369    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790f31ca-211c-4e39-9ab3-6158bbe697b6-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.844678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:38.844384    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhr79\" (UniqueName: \"kubernetes.io/projected/790f31ca-211c-4e39-9ab3-6158bbe697b6-kube-api-access-bhr79\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:39.086484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.086447    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"]
Oct 30 22:16:39.086910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.086897    4410 cpu_manager.go:414] "containerMap: removing container" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.086971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.086911    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.086971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.086923    4410 cpu_manager.go:414] "containerMap: removing container" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.086971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.086929    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.087065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.087009    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.087065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.087017    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerName="istio-proxy"
Oct 30 22:16:39.090359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.090343    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.142001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.141978    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:39.142104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.142052    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:39.142104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.142088    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:16:39.144880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.144867    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:39.148638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.148619    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"]
Oct 30 22:16:39.248584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.248548    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.248805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.248611    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrv4n\" (UniqueName: \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.248805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.248740    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.350175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.350088    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrv4n\" (UniqueName: \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.350175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.350161    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.350409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.350304    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.350975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.350948    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.352872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.352850    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.357716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.357694    4410 generic.go:356] "Generic (PLEG): container finished" podID="790f31ca-211c-4e39-9ab3-6158bbe697b6" containerID="86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790" exitCode=0
Oct 30 22:16:39.357823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.357754    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"
Oct 30 22:16:39.357871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.357820    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerDied","Data":"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"}
Oct 30 22:16:39.357908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.357882    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s" event={"ID":"790f31ca-211c-4e39-9ab3-6158bbe697b6","Type":"ContainerDied","Data":"ef7e7d45c1db0c6000604ec8b3c5101c9dd01cd6036d4572f0801d999e02f096"}
Oct 30 22:16:39.357945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.357914    4410 scope.go:122] "RemoveContainer" containerID="86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"
Oct 30 22:16:39.368893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.368873    4410 scope.go:122] "RemoveContainer" containerID="f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"
Oct 30 22:16:39.377064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.377048    4410 scope.go:122] "RemoveContainer" containerID="86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"
Oct 30 22:16:39.377329 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:16:39.377311    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790\": container with ID starting with 86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790 not found: ID does not exist" containerID="86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"
Oct 30 22:16:39.377402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.377339    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790"} err="failed to get container status \"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790\": rpc error: code = NotFound desc = could not find container \"86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790\": container with ID starting with 86741d9ceedd0f6828ffbca5b7454190db72c277fff92a79e4656dd6f909b790 not found: ID does not exist"
Oct 30 22:16:39.377402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.377365    4410 scope.go:122] "RemoveContainer" containerID="f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"
Oct 30 22:16:39.377601 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:16:39.377581    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617\": container with ID starting with f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617 not found: ID does not exist" containerID="f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"
Oct 30 22:16:39.377643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.377607    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617"} err="failed to get container status \"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617\": rpc error: code = NotFound desc = could not find container \"f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617\": container with ID starting with f1f07ac0aa4b5543d672e50750586a8782832eaaaa4f0c0d66e1c250937ce617 not found: ID does not exist"
Oct 30 22:16:39.406757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.406726    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrv4n\" (UniqueName: \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.522604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.522573    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:16:39.524759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.524740    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-97ca39-openshift-default-759c9cfdb7-k4g5s"]
Oct 30 22:16:39.703494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.703463    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:39.988924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:39.988836    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:40.084174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:40.084150    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"]
Oct 30 22:16:40.086278 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:16:40.086242    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b99b4d2_78e7_4905_8605_72cea5b75a18.slice/crio-0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2 WatchSource:0}: Error finding container 0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2: Status 404 returned error can't find the container with id 0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2
Oct 30 22:16:40.365984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:40.365952    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd" event={"ID":"2b99b4d2-78e7-4905-8605-72cea5b75a18","Type":"ContainerStarted","Data":"0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2"}
Oct 30 22:16:40.558253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:40.558168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:41.025766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:41.025730    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790f31ca-211c-4e39-9ab3-6158bbe697b6" path="/var/lib/kubelet/pods/790f31ca-211c-4e39-9ab3-6158bbe697b6/volumes"
Oct 30 22:16:41.374088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:41.374006    4410 generic.go:356] "Generic (PLEG): container finished" podID="2b99b4d2-78e7-4905-8605-72cea5b75a18" containerID="4256a1711d550896f1c8c466f5b976b18ac83abeaa88cade3646a2bb460c48e9" exitCode=0
Oct 30 22:16:41.374088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:41.374044    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd" event={"ID":"2b99b4d2-78e7-4905-8605-72cea5b75a18","Type":"ContainerDied","Data":"4256a1711d550896f1c8c466f5b976b18ac83abeaa88cade3646a2bb460c48e9"}
Oct 30 22:16:42.520413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.520386    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:42.595941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.595913    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\" (UniqueName: \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\") pod \"2b99b4d2-78e7-4905-8605-72cea5b75a18\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") "
Oct 30 22:16:42.596079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.595971    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\") pod \"2b99b4d2-78e7-4905-8605-72cea5b75a18\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") "
Oct 30 22:16:42.596079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.596040    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\") pod \"2b99b4d2-78e7-4905-8605-72cea5b75a18\" (UID: \"2b99b4d2-78e7-4905-8605-72cea5b75a18\") "
Oct 30 22:16:42.596439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.596416    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca" pod "2b99b4d2-78e7-4905-8605-72cea5b75a18" (UID: "2b99b4d2-78e7-4905-8605-72cea5b75a18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:42.598097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.598072    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca" pod "2b99b4d2-78e7-4905-8605-72cea5b75a18" (UID: "2b99b4d2-78e7-4905-8605-72cea5b75a18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:42.598097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.598073    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n" pod "2b99b4d2-78e7-4905-8605-72cea5b75a18" (UID: "2b99b4d2-78e7-4905-8605-72cea5b75a18"). InnerVolumeSpecName "kube-api-access-xrv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.697619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.697586    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:42.697619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.697616    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrv4n\" (UniqueName: \"kubernetes.io/projected/2b99b4d2-78e7-4905-8605-72cea5b75a18-kube-api-access-xrv4n\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:42.697619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:42.697626    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b99b4d2-78e7-4905-8605-72cea5b75a18-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.387184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:43.387103    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"
Oct 30 22:16:43.387340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:43.387098    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd" event={"ID":"2b99b4d2-78e7-4905-8605-72cea5b75a18","Type":"ContainerDied","Data":"0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2"}
Oct 30 22:16:43.387340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:43.387234    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0510510b19cba6c125b13866ceda38c12fd05fa36cdfcabd9b52621203e903a2"
Oct 30 22:16:43.522912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:43.522873    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.545240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:48.545191    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:50.521455 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:16:50.521416    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:16:52.772819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:52.772510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:53.304162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:53.304119    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:53.470474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:53.470434    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.371252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:54.371204    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.595720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:54.595675    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.763794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:54.762373    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.938970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:54.938932    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"]
Oct 30 22:16:54.987876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:54.987843    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-859078-wk2xd"]
Oct 30 22:16:55.018829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:55.018751    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b99b4d2-78e7-4905-8605-72cea5b75a18" path="/var/lib/kubelet/pods/2b99b4d2-78e7-4905-8605-72cea5b75a18/volumes"
Oct 30 22:16:57.509476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.509424    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.906648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.906606    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"]
Oct 30 22:16:57.907300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.907266    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2b99b4d2-78e7-4905-8605-72cea5b75a18" containerName="revoke-keys"
Oct 30 22:16:57.907300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.907300    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b99b4d2-78e7-4905-8605-72cea5b75a18" containerName="revoke-keys"
Oct 30 22:16:57.907508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.907456    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b99b4d2-78e7-4905-8605-72cea5b75a18" containerName="revoke-keys"
Oct 30 22:16:57.910904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.910883    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:57.963574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:57.963526    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:58.012736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.012706    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.012954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.012934    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:58.013045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.012994    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:16:58.013045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.013038    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:58.013235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.013216    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.013355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.013338    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4zx\" (UniqueName: \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.016956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.016933    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"]
Oct 30 22:16:58.114382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.114343    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4zx\" (UniqueName: \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.114576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.114422    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.114576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.114511    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.115094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.115071    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.117086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.117058    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.185544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.185427    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4zx\" (UniqueName: \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\") pod \"maas-api-revoke-sub-7779b9d3-2b2th\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.224917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.224883    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:16:58.569493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.569406    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:58.643102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:58.643076    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"]
Oct 30 22:16:58.646654 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:16:58.646625    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2d953a_7cb9_41ba_83ba_364d2e70b404.slice/crio-6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b WatchSource:0}: Error finding container 6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b: Status 404 returned error can't find the container with id 6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b
Oct 30 22:16:59.507831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:59.507585    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th" event={"ID":"ab2d953a-7cb9-41ba-83ba-364d2e70b404","Type":"ContainerStarted","Data":"44a4377f370ed0efaa02102c2fb6409da7e0c72c0fbfb04cc70fea4dad3d2af1"}
Oct 30 22:16:59.507831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:59.507667    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th" event={"ID":"ab2d953a-7cb9-41ba-83ba-364d2e70b404","Type":"ContainerStarted","Data":"6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b"}
Oct 30 22:16:59.621934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:59.621871    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th" podStartSLOduration=2.621854566 podStartE2EDuration="2.621854566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:59.621507459 +0000 UTC m=+2072.710245263" watchObservedRunningTime="2026-10-30 22:16:59.621854566 +0000 UTC m=+2072.710592370"
Oct 30 22:16:59.858390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:16:59.858359    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:00.515298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:00.515257    4410 generic.go:356] "Generic (PLEG): container finished" podID="ab2d953a-7cb9-41ba-83ba-364d2e70b404" containerID="44a4377f370ed0efaa02102c2fb6409da7e0c72c0fbfb04cc70fea4dad3d2af1" exitCode=0
Oct 30 22:17:00.515493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:00.515299    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th" event={"ID":"ab2d953a-7cb9-41ba-83ba-364d2e70b404","Type":"ContainerDied","Data":"44a4377f370ed0efaa02102c2fb6409da7e0c72c0fbfb04cc70fea4dad3d2af1"}
Oct 30 22:17:01.444511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.444477    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": read tcp 10.133.0.2:51420->10.133.0.128:15021: read: connection reset by peer" start-of-body=
Oct 30 22:17:01.444931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.444548    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": read tcp 10.133.0.2:51420->10.133.0.128:15021: read: connection reset by peer"
Oct 30 22:17:01.522331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.522302    4410 generic.go:356] "Generic (PLEG): container finished" podID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerID="617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a" exitCode=0
Oct 30 22:17:01.522468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.522371    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerDied","Data":"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"}
Oct 30 22:17:01.523130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.522902    4410 scope.go:122] "RemoveContainer" containerID="617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"
Oct 30 22:17:01.524396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.524366    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:01.524496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.524451    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:01.524565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.524498    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:01.679145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.679118    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:17:01.865873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.865794    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\" (UniqueName: \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\") pod \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") "
Oct 30 22:17:01.865873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.865860    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\") pod \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") "
Oct 30 22:17:01.866052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.865909    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\") pod \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\" (UID: \"ab2d953a-7cb9-41ba-83ba-364d2e70b404\") "
Oct 30 22:17:01.866413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.866384    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca" pod "ab2d953a-7cb9-41ba-83ba-364d2e70b404" (UID: "ab2d953a-7cb9-41ba-83ba-364d2e70b404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:01.867997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.867969    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx" pod "ab2d953a-7cb9-41ba-83ba-364d2e70b404" (UID: "ab2d953a-7cb9-41ba-83ba-364d2e70b404"). InnerVolumeSpecName "kube-api-access-bf4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:01.868087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.867980    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca" pod "ab2d953a-7cb9-41ba-83ba-364d2e70b404" (UID: "ab2d953a-7cb9-41ba-83ba-364d2e70b404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:01.966819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.966763    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:01.966819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.966820    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf4zx\" (UniqueName: \"kubernetes.io/projected/ab2d953a-7cb9-41ba-83ba-364d2e70b404-kube-api-access-bf4zx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:01.967047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:01.966835    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab2d953a-7cb9-41ba-83ba-364d2e70b404-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:02.186543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.186509    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:02.367798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.362674    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:02.521268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.521180    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:02.529219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.529189    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"
Oct 30 22:17:02.529219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.529207    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th" event={"ID":"ab2d953a-7cb9-41ba-83ba-364d2e70b404","Type":"ContainerDied","Data":"6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b"}
Oct 30 22:17:02.529412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.529259    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d70004558b34474096b92f81715cb3672dec265f80ee847ddc03390aa8d303b"
Oct 30 22:17:02.531326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:02.531293    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerStarted","Data":"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"}
Oct 30 22:17:03.035434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:03.035404    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:17:03.035641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:03.035532    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:17:03.035641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:03.035570    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 22:17:04.035889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:04.035840    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:17:04.036365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:04.035914    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 22:17:05.032021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.031985    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.036445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.036411    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.036817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.036485    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 22:17:05.302305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.302203    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.497255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.497223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.675251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.675208    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.897543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.897507    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:17:05.897998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.897982    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ab2d953a-7cb9-41ba-83ba-364d2e70b404" containerName="revoke-keys"
Oct 30 22:17:05.898059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.897999    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2d953a-7cb9-41ba-83ba-364d2e70b404" containerName="revoke-keys"
Oct 30 22:17:05.898114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.898105    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2d953a-7cb9-41ba-83ba-364d2e70b404" containerName="revoke-keys"
Oct 30 22:17:05.901380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.901362    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:05.908210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.908177    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26k2k\" (UniqueName: \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:05.908284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.908267    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:05.958328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:05.958240    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a28730\""
Oct 30 22:17:06.008535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.008510    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a28730-dockercfg-7xbwd\""
Oct 30 22:17:06.009340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.009300    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26k2k\" (UniqueName: \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:06.012218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.012197    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:17:06.013888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.013857    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:06.018410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.014579    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:06.049804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.044049    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.049804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.044105    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 22:17:06.066735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.066689    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:17:06.088800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.086942    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.090912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.090848    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26k2k\" (UniqueName: \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\") pod \"payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:06.111484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.111460    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:17:06.114678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.114652    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkjw\" (UniqueName: \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.114845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.114829    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.213935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.213859    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:06.216562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.216289    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkjw\" (UniqueName: \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.216562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.216420    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.217134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.217114    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.274800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.274747    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkjw\" (UniqueName: \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\") pod \"payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.407519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.407481    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:06.620625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.620599    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:17:06.622946 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:17:06.622912    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f74f11_5b83_44fa_b1a3_9960a81b4cb0.slice/crio-068f5449f2b2a50dfc28fb552a7cd10026f847dc10f511a094bb9e70a9d9380c WatchSource:0}: Error finding container 068f5449f2b2a50dfc28fb552a7cd10026f847dc10f511a094bb9e70a9d9380c: Status 404 returned error can't find the container with id 068f5449f2b2a50dfc28fb552a7cd10026f847dc10f511a094bb9e70a9d9380c
Oct 30 22:17:06.807929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:06.807904    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:17:06.810339 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:17:06.810305    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1325f13b_93df_4e61_80fc_4ca8f59e9c5a.slice/crio-54b6d97fe9c6cca0373e7f4d8d50860b9375be659ab2e278f717d296de36f26f WatchSource:0}: Error finding container 54b6d97fe9c6cca0373e7f4d8d50860b9375be659ab2e278f717d296de36f26f: Status 404 returned error can't find the container with id 54b6d97fe9c6cca0373e7f4d8d50860b9375be659ab2e278f717d296de36f26f
Oct 30 22:17:07.036660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.036626    4410 patch_prober.go:29] interesting pod/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.036849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.036675    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 22:17:07.586404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.586364    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" event={"ID":"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0","Type":"ContainerStarted","Data":"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"}
Oct 30 22:17:07.586932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.586431    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" event={"ID":"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0","Type":"ContainerStarted","Data":"068f5449f2b2a50dfc28fb552a7cd10026f847dc10f511a094bb9e70a9d9380c"}
Oct 30 22:17:07.586932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.586570    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:07.588109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.588078    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" event={"ID":"1325f13b-93df-4e61-80fc-4ca8f59e9c5a","Type":"ContainerStarted","Data":"6ba6d7c6c495dd633f97d9a15c1fd8a549c3d70e48817d2e48cb298a6208c44a"}
Oct 30 22:17:07.588222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.588125    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" event={"ID":"1325f13b-93df-4e61-80fc-4ca8f59e9c5a","Type":"ContainerStarted","Data":"54b6d97fe9c6cca0373e7f4d8d50860b9375be659ab2e278f717d296de36f26f"}
Oct 30 22:17:07.588222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.588146    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:07.701756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.701697    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" podStartSLOduration=2.701683152 podStartE2EDuration="2.701683152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.699500386 +0000 UTC m=+2080.788238199" watchObservedRunningTime="2026-10-30 22:17:07.701683152 +0000 UTC m=+2080.790420955"
Oct 30 22:17:07.813801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:07.812822    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" podStartSLOduration=1.812804162 podStartE2EDuration="1.812804162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.811169434 +0000 UTC m=+2080.899907238" watchObservedRunningTime="2026-10-30 22:17:07.812804162 +0000 UTC m=+2080.901541966"
Oct 30 22:17:08.040093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:08.040065    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:17:08.040455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:08.040431    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:17:08.041000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:08.040984    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:17:08.781522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:08.781480    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.067432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:10.067398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.438738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:11.438702    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.856200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:11.856122    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.035315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:12.035277    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.250453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:12.250419    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.076924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:13.076890    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.321947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:13.321913    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.899478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:13.899445    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.154329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.154218    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.607705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.607593    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.808455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.808416    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:17:14.812402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.812379    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.876036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.875957    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d66ae6-openshift-default-dockercfg-qxcxt\""
Oct 30 22:17:14.918119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.918085    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:17:14.969473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969431    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9sl\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969491    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969545    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969583    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969659    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969710    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969850    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969905    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.969957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.969948    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:14.974900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:14.974813    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.071341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071312    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071409    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071463    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071509    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071568    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9sl\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071605    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071661    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071721    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071753    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.071947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.071893    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.072093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.072071    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.072163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.072131    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.072404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.072382    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.072497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.072403    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.073817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.073752    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.074070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.074053    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.128447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.128244    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.128593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.128468    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9sl\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\") pod \"e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") " pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.138065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.137990    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.313321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.313285    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.428631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.428597    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:15.586895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.586858    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.859636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.859611    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:17:15.861341 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:17:15.861315    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580c0cc4_6f99_47de_9070_d707b8a5fa85.slice/crio-31fd7d3772b3002b9d8d5516604bcfee403c5c9148a8afe3c6bbaf3a5f9e8aa0 WatchSource:0}: Error finding container 31fd7d3772b3002b9d8d5516604bcfee403c5c9148a8afe3c6bbaf3a5f9e8aa0: Status 404 returned error can't find the container with id 31fd7d3772b3002b9d8d5516604bcfee403c5c9148a8afe3c6bbaf3a5f9e8aa0
Oct 30 22:17:15.864284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.864236    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:15.864397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.864326    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:15.864397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.864374    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:15.866195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:15.866162    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.648588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:16.648552    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerStarted","Data":"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"}
Oct 30 22:17:16.649011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:16.648635    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerStarted","Data":"31fd7d3772b3002b9d8d5516604bcfee403c5c9148a8afe3c6bbaf3a5f9e8aa0"}
Oct 30 22:17:16.767274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:16.767215    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podStartSLOduration=2.767197188 podStartE2EDuration="2.767197188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:16.76411186 +0000 UTC m=+2089.852849663" watchObservedRunningTime="2026-10-30 22:17:16.767197188 +0000 UTC m=+2089.855934991"
Oct 30 22:17:17.429629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:17.429598    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:17.435205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:17.435177    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:17.654521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:17.654493    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:17.655345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:17.655327    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:17:18.597618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:18.597588    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:17:18.597853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:18.597660    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:17:19.850020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:19.849985    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.031587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:20.031546    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.205654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:20.205615    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.522417 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:17:20.522332    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:17:20.543600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:20.543566    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.785210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:20.785129    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.789831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:22.789764    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.126654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:23.126567    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.302300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:23.302265    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:25.539783 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:25.539736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.142441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:28.142406    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.307323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:28.307283    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.472449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:28.472351    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:33.727206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:33.727170    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.206365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:38.206319    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:38.434815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:38.434753    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:40.829257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:40.829166    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.672321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:45.672273    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.831831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:45.831759    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.521540 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:17:50.521500    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:17:50.538269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:50.538221    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.437618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:17:55.437581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.199561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:00.199524    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.363648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:00.363608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.545284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:00.545171    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:05.567913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:05.567877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:09.115183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:09.115142    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:09.634922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:09.634886    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.289056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:12.289020    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:17.430191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:17.430150    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:20.522471 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:18:20.522433    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:18:23.917985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:23.917943    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:24.147730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:24.147677    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:24.353573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:24.353475    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.873220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:25.873177    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:28.212883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:28.212850    4410 scope.go:122] "RemoveContainer" containerID="e306a8330431751bf71a346a6e7badcecde9e560ef54eb14e029d81f5028f15b"
Oct 30 22:18:28.222277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:28.222252    4410 scope.go:122] "RemoveContainer" containerID="61eedaf90d5f4bdb551316efe8fe5a466f259f590dd221204bcaa9a2ebfbee4d"
Oct 30 22:18:28.230352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:28.230331    4410 scope.go:122] "RemoveContainer" containerID="5c1b8f4ac91f09646adc0b1efe6440073ebf7f4301e47a2de2066514bf3f04a6"
Oct 30 22:18:28.887904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:28.887871    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:29.047528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:29.047493    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:29.351892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:29.351858    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:34.092635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:34.092602    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.236743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.236707    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.482228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.482186    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.903438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.903404    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:18:39.908712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.908686    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980125    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980180    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980206    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980232    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980309    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980334    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980404    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980463    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg69l\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:39.980796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:39.980561    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.036984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.036936    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:40.045426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.045344    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:18:40.056437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.056042    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8b1e0a-openshift-default-dockercfg-v8tpp\""
Oct 30 22:18:40.091238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.091148    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg69l\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.091997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.091706    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.092630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.092447    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.092889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.092845    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.092987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.092910    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.092987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.092938    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.092987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.092961    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.093223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.093093    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.093223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.093122    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.093223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.093209    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.093386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.093220    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.094080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.094058    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.094216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.094104    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.094329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.094256    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.098848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.098351    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.098848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.098431    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.158031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.157951    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg69l\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.158614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.158592    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\") pod \"e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") " pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.225119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.225091    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:40.668264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.668102    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:40.669997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.669973    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:18:40.672145 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:18:40.672121    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba18dc36_4920_49b2_a111_498e08aa3344.slice/crio-d214b535b78eef1db4a98dadd2fabab0c7718154b03a4d23ad393a68063d2e46 WatchSource:0}: Error finding container d214b535b78eef1db4a98dadd2fabab0c7718154b03a4d23ad393a68063d2e46: Status 404 returned error can't find the container with id d214b535b78eef1db4a98dadd2fabab0c7718154b03a4d23ad393a68063d2e46
Oct 30 22:18:40.675029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.675000    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:40.675101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.675062    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:40.675101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:40.675091    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:41.398277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:41.398239    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerStarted","Data":"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"}
Oct 30 22:18:41.398456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:41.398294    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerStarted","Data":"d214b535b78eef1db4a98dadd2fabab0c7718154b03a4d23ad393a68063d2e46"}
Oct 30 22:18:41.519459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:41.517975    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podStartSLOduration=2.517957527 podStartE2EDuration="2.517957527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:41.516423863 +0000 UTC m=+2174.605161665" watchObservedRunningTime="2026-10-30 22:18:41.517957527 +0000 UTC m=+2174.606695330"
Oct 30 22:18:42.225950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:42.225917    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:42.231443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:42.231418    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:42.404609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:42.404574    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:42.405537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:42.405516    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:18:42.858642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:42.858610    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:43.385638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:43.385579    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.312763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:44.312739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.490115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:44.490080    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.667193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:44.667163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:46.632325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:46.632281    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.778159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:47.778125    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.995854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:47.995825    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:48.164006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.159716    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:18:48.166303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.165317    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.217967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.217918    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8b1e0a\""
Oct 30 22:18:48.222179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.222149    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:18:48.240825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.240797    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.240965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.240877    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krv58\" (UniqueName: \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.344303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.344236    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.359501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.359469    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krv58\" (UniqueName: \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.360588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.360398    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.373415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.373382    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:18:48.404046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.404013    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:18:48.404432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.404413    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.428697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.428667    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krv58\" (UniqueName: \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.474212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.474185    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8b1e0a-dockercfg-jwqhb\""
Oct 30 22:18:48.500338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.500306    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:48.578805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.577618    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.578805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.577848    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smtx\" (UniqueName: \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.679055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.678956    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smtx\" (UniqueName: \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.679055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.679032    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.679759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.679735    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.736524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.736491    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smtx\" (UniqueName: \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\") pod \"payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:48.910141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:48.910115    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:18:48.912335 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:18:48.912307    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ac0830_d68d_4a76_9d6e_401e274169df.slice/crio-5ec4d6cf76925d11db5942d9e70b5d55f5918e8f09f49f722a3b9478f6fe9920 WatchSource:0}: Error finding container 5ec4d6cf76925d11db5942d9e70b5d55f5918e8f09f49f722a3b9478f6fe9920: Status 404 returned error can't find the container with id 5ec4d6cf76925d11db5942d9e70b5d55f5918e8f09f49f722a3b9478f6fe9920
Oct 30 22:18:49.025672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.025643    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:49.427998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.427968    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:18:49.429558 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:18:49.429530    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb750bfe7_3374_4051_9f07_d585b615eebb.slice/crio-d166d859c1c3fdde6a5a8ca0c0b5b3fd06869ce6f84384227ef5fdde9fcf1fba WatchSource:0}: Error finding container d166d859c1c3fdde6a5a8ca0c0b5b3fd06869ce6f84384227ef5fdde9fcf1fba: Status 404 returned error can't find the container with id d166d859c1c3fdde6a5a8ca0c0b5b3fd06869ce6f84384227ef5fdde9fcf1fba
Oct 30 22:18:49.478501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.478466    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" event={"ID":"b750bfe7-3374-4051-9f07-d585b615eebb","Type":"ContainerStarted","Data":"d166d859c1c3fdde6a5a8ca0c0b5b3fd06869ce6f84384227ef5fdde9fcf1fba"}
Oct 30 22:18:49.480338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.480312    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" event={"ID":"86ac0830-d68d-4a76-9d6e-401e274169df","Type":"ContainerStarted","Data":"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"}
Oct 30 22:18:49.480453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.480355    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" event={"ID":"86ac0830-d68d-4a76-9d6e-401e274169df","Type":"ContainerStarted","Data":"5ec4d6cf76925d11db5942d9e70b5d55f5918e8f09f49f722a3b9478f6fe9920"}
Oct 30 22:18:49.480522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.480485    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:18:49.596722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:49.596671    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" podStartSLOduration=1.596657931 podStartE2EDuration="1.596657931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:49.594188745 +0000 UTC m=+2182.682926547" watchObservedRunningTime="2026-10-30 22:18:49.596657931 +0000 UTC m=+2182.685395733"
Oct 30 22:18:50.488058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:50.488011    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" event={"ID":"b750bfe7-3374-4051-9f07-d585b615eebb","Type":"ContainerStarted","Data":"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"}
Oct 30 22:18:50.488586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:50.488076    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:18:50.522367 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:18:50.522322    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:18:50.605414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:50.605342    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" podStartSLOduration=2.605322454 podStartE2EDuration="2.605322454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:50.601533487 +0000 UTC m=+2183.690271285" watchObservedRunningTime="2026-10-30 22:18:50.605322454 +0000 UTC m=+2183.694060258"
Oct 30 22:18:53.665509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:53.665465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:55.517751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.517716    4410 generic.go:356] "Generic (PLEG): container finished" podID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerID="49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57" exitCode=0
Oct 30 22:18:55.518208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.517801    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerDied","Data":"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"}
Oct 30 22:18:55.518208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.518167    4410 scope.go:122] "RemoveContainer" containerID="49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"
Oct 30 22:18:55.519598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.519571    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:55.519657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.519636    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:55.519693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:55.519667    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:18:56.525698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:56.525665    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerStarted","Data":"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"}
Oct 30 22:18:57.429474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:57.429446    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:18:57.429659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:57.429620    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:18:57.429705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:57.429662    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:18:58.429796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:58.429747    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:18:58.430172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:58.429824    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:18:58.536727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:58.536696    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:59.429362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:59.429328    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:18:59.429540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:18:59.429377    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:19:00.270298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.270249    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"]
Oct 30 22:19:00.274563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.274540    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.324322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.324295    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.324449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.324351    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.324497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.324483    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjq6\" (UniqueName: \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.330618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.330594    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"]
Oct 30 22:19:00.376846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.376764    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:19:00.377018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.376903    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:00.377018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.377005    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:00.377154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.377064    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:00.425316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.425288    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.425446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.425338    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.425446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.425397    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjq6\" (UniqueName: \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.426248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.426223    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.427539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.427514    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.429657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.429638    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:19:00.429715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.429691    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:19:00.482145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.482111    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjq6\" (UniqueName: \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.491484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.491460    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:19:00.588412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.588335    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:00.987746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:00.987717    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"]
Oct 30 22:19:00.988719 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:19:00.988684    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cdd32e_04e6_4dfb_aa56_d8e9c11df047.slice/crio-c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52 WatchSource:0}: Error finding container c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52: Status 404 returned error can't find the container with id c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52
Oct 30 22:19:01.429641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.429608    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:19:01.429996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.429660    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:19:01.497447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.497412    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:19:01.579844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.579810    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c" event={"ID":"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047","Type":"ContainerStarted","Data":"50337cf669d922be755b82aaa81d1c16325e1c1f092a9faade80ebb658c84cc9"}
Oct 30 22:19:01.580011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.579866    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c" event={"ID":"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047","Type":"ContainerStarted","Data":"c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52"}
Oct 30 22:19:01.728434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:01.728364    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c" podStartSLOduration=1.7283454040000001 podStartE2EDuration="1.728345404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:01.727646668 +0000 UTC m=+2194.816384472" watchObservedRunningTime="2026-10-30 22:19:01.728345404 +0000 UTC m=+2194.817083214"
Oct 30 22:19:02.434398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:02.434368    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:19:02.434793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:02.434640    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:19:02.435444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:02.435423    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:19:02.586589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:02.586557    4410 generic.go:356] "Generic (PLEG): container finished" podID="c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" containerID="50337cf669d922be755b82aaa81d1c16325e1c1f092a9faade80ebb658c84cc9" exitCode=0
Oct 30 22:19:02.586731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:02.586636    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c" event={"ID":"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047","Type":"ContainerDied","Data":"50337cf669d922be755b82aaa81d1c16325e1c1f092a9faade80ebb658c84cc9"}
Oct 30 22:19:03.736966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.736943    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:03.856050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.855979    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:03.873544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.873511    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\") pod \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") "
Oct 30 22:19:03.874000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.873976    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\") pod \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") "
Oct 30 22:19:03.874270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.874250    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\" (UniqueName: \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\") pod \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\" (UID: \"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047\") "
Oct 30 22:19:03.874374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.874355    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca" pod "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" (UID: "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:03.874923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.874900    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:03.877636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.877612    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6" pod "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" (UID: "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047"). InnerVolumeSpecName "kube-api-access-zbjq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:03.877978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.877908    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca" pod "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" (UID: "c3cdd32e-04e6-4dfb-aa56-d8e9c11df047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:03.976683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.976650    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:03.976683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:03.976680    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjq6\" (UniqueName: \"kubernetes.io/projected/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047-kube-api-access-zbjq6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:04.600619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:04.600589    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"
Oct 30 22:19:04.600810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:04.600616    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c" event={"ID":"c3cdd32e-04e6-4dfb-aa56-d8e9c11df047","Type":"ContainerDied","Data":"c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52"}
Oct 30 22:19:04.600810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:04.600668    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57bc4d88f59fb22d8ec039e2e03161741b7f4adab04e3ef0dacac58c443bd52"
Oct 30 22:19:08.981999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:08.981953    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:13.857479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:13.857398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:15.294230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:15.294189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.121198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:16.121163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:18.869759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:18.869714    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.417847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:20.417803    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.521146 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:19:20.521113    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:19:20.599846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:20.599816    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:19:20.705928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:20.705852    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:19:20.984965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:20.984878    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.161692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:21.161650    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:25.601068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.600990    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" podUID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" containerName="payload-processing" containerID="cri-o://6ba6d7c6c495dd633f97d9a15c1fd8a549c3d70e48817d2e48cb298a6208c44a" gracePeriod=25
Oct 30 22:19:25.706486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.706430    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" podUID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" containerName="payload-pre-processing" containerID="cri-o://94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde" gracePeriod=25
Oct 30 22:19:25.724105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.724081    4410 generic.go:356] "Generic (PLEG): container finished" podID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" containerID="6ba6d7c6c495dd633f97d9a15c1fd8a549c3d70e48817d2e48cb298a6208c44a" exitCode=0
Oct 30 22:19:25.724199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.724118    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" event={"ID":"1325f13b-93df-4e61-80fc-4ca8f59e9c5a","Type":"ContainerDied","Data":"6ba6d7c6c495dd633f97d9a15c1fd8a549c3d70e48817d2e48cb298a6208c44a"}
Oct 30 22:19:25.876348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.876319    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:19:25.934013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.933977    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\") pod \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") "
Oct 30 22:19:25.934168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.934085    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\" (UniqueName: \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\") pod \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\" (UID: \"1325f13b-93df-4e61-80fc-4ca8f59e9c5a\") "
Oct 30 22:19:25.934426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.934401    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume" pod "1325f13b-93df-4e61-80fc-4ca8f59e9c5a" (UID: "1325f13b-93df-4e61-80fc-4ca8f59e9c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:25.934735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.934713    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.936575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.936551    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw" pod "1325f13b-93df-4e61-80fc-4ca8f59e9c5a" (UID: "1325f13b-93df-4e61-80fc-4ca8f59e9c5a"). InnerVolumeSpecName "kube-api-access-xqkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:25.956894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:25.956871    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:19:26.035447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.035418    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\") pod \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") "
Oct 30 22:19:26.035616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.035506    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\" (UniqueName: \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\") pod \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\" (UID: \"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0\") "
Oct 30 22:19:26.035878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.035863    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkjw\" (UniqueName: \"kubernetes.io/projected/1325f13b-93df-4e61-80fc-4ca8f59e9c5a-kube-api-access-xqkjw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:26.035931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.035863    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume" pod "a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" (UID: "a7f74f11-5b83-44fa-b1a3-9960a81b4cb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:26.037590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.037566    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k" pod "a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" (UID: "a7f74f11-5b83-44fa-b1a3-9960a81b4cb0"). InnerVolumeSpecName "kube-api-access-26k2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:26.092479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.092450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.137426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.137342    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26k2k\" (UniqueName: \"kubernetes.io/projected/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-kube-api-access-26k2k\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:26.137426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.137380    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:26.250578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.250545    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.435475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.435442    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.739506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.739415    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg" event={"ID":"1325f13b-93df-4e61-80fc-4ca8f59e9c5a","Type":"ContainerDied","Data":"54b6d97fe9c6cca0373e7f4d8d50860b9375be659ab2e278f717d296de36f26f"}
Oct 30 22:19:26.739506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.739440    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"
Oct 30 22:19:26.740047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.739511    4410 scope.go:122] "RemoveContainer" containerID="6ba6d7c6c495dd633f97d9a15c1fd8a549c3d70e48817d2e48cb298a6208c44a"
Oct 30 22:19:26.741300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.741274    4410 generic.go:356] "Generic (PLEG): container finished" podID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" containerID="94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde" exitCode=0
Oct 30 22:19:26.741436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.741312    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" event={"ID":"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0","Type":"ContainerDied","Data":"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"}
Oct 30 22:19:26.741436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.741324    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"
Oct 30 22:19:26.741436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.741347    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx" event={"ID":"a7f74f11-5b83-44fa-b1a3-9960a81b4cb0","Type":"ContainerDied","Data":"068f5449f2b2a50dfc28fb552a7cd10026f847dc10f511a094bb9e70a9d9380c"}
Oct 30 22:19:26.750608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.750590    4410 scope.go:122] "RemoveContainer" containerID="94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"
Oct 30 22:19:26.759234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.759215    4410 scope.go:122] "RemoveContainer" containerID="94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"
Oct 30 22:19:26.759462 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:19:26.759445    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde\": container with ID starting with 94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde not found: ID does not exist" containerID="94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"
Oct 30 22:19:26.759508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.759469    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde"} err="failed to get container status \"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde\": rpc error: code = NotFound desc = could not find container \"94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde\": container with ID starting with 94b848ebf6d2f0ee2a51aa0074be694cccbec5c7b5f2d96c602cd59ee308cdde not found: ID does not exist"
Oct 30 22:19:26.903708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.903676    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:19:26.906907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:26.906886    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-7c7fc89749-mpmmg"]
Oct 30 22:19:27.020182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:27.020113    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" path="/var/lib/kubelet/pods/1325f13b-93df-4e61-80fc-4ca8f59e9c5a/volumes"
Oct 30 22:19:27.059894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:27.059863    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:19:27.064951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:27.064922    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-59d784d8c-n7rfx"]
Oct 30 22:19:28.265123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:28.265095    4410 scope.go:122] "RemoveContainer" containerID="9fca73136863da1c77e2c799b6a1bf27ea7caec9c73142c5e4afb83b43ecfef6"
Oct 30 22:19:29.019525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:29.019494    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" path="/var/lib/kubelet/pods/a7f74f11-5b83-44fa-b1a3-9960a81b4cb0/volumes"
Oct 30 22:19:29.426788 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:29.426745    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:19:29.427192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:29.426946    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" containerID="cri-o://3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061" gracePeriod=30
Oct 30 22:19:30.619997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:30.619956    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.298117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:31.298079    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.450899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:31.450869    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:32.208627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.208590    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:32.209025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.208647    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:32.386450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.386415    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:32.603188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.603120    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:32.761148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.761113    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:32.894970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.894933    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"]
Oct 30 22:19:32.944004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:32.943973    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a54db-nhq7c"]
Oct 30 22:19:33.019733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:33.019699    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" path="/var/lib/kubelet/pods/c3cdd32e-04e6-4dfb-aa56-d8e9c11df047/volumes"
Oct 30 22:19:33.343756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:33.343670    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:34.689944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.689882    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:19:34.735878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.735836    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.735948    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736008    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736078    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:34.736156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736117    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736156    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736244    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736318    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736338    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:34.736476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736365    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736400    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\") pod \"e71c670c-8115-4339-9481-7ac72c34920f\" (UID: \"e71c670c-8115-4339-9481-7ac72c34920f\") "
Oct 30 22:19:34.736604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736570    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:34.736604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736590    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:34.736869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736843    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:34.737006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.736985    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.737085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.737012    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.737085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.737028    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.737085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.737042    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71c670c-8115-4339-9481-7ac72c34920f-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.737085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.737056    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.738431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.738403    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:34.738536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.738427    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:34.738536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.738461    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "kube-api-access-hdcmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:34.738536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.738471    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo" pod "e71c670c-8115-4339-9481-7ac72c34920f" (UID: "e71c670c-8115-4339-9481-7ac72c34920f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:34.790607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.790576    4410 generic.go:356] "Generic (PLEG): container finished" podID="e71c670c-8115-4339-9481-7ac72c34920f" containerID="3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061" exitCode=0
Oct 30 22:19:34.790712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.790616    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"
Oct 30 22:19:34.790712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.790656    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" event={"ID":"e71c670c-8115-4339-9481-7ac72c34920f","Type":"ContainerDied","Data":"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"}
Oct 30 22:19:34.790712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.790707    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj" event={"ID":"e71c670c-8115-4339-9481-7ac72c34920f","Type":"ContainerDied","Data":"1283f36f21b3f842d98e3199446ced800f6eff2bd9f99c16757b07ecd3428f31"}
Oct 30 22:19:34.790869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.790731    4410 scope.go:122] "RemoveContainer" containerID="3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"
Oct 30 22:19:34.803535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.803514    4410 scope.go:122] "RemoveContainer" containerID="3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"
Oct 30 22:19:34.804035 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:19:34.804005    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061\": container with ID starting with 3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061 not found: ID does not exist" containerID="3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"
Oct 30 22:19:34.804128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.804046    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061"} err="failed to get container status \"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061\": rpc error: code = NotFound desc = could not find container \"3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061\": container with ID starting with 3ad70b94a8efc91b190ca20323eecc2071ec52453f8d7e624c34aa2d4bce6061 not found: ID does not exist"
Oct 30 22:19:34.837896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.837874    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71c670c-8115-4339-9481-7ac72c34920f-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.837896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.837895    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdcmx\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-kube-api-access-hdcmx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.838073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.837905    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71c670c-8115-4339-9481-7ac72c34920f-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.838073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.837919    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71c670c-8115-4339-9481-7ac72c34920f-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.965083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.964981    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:19:34.966174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:34.966150    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-j22vj"]
Oct 30 22:19:35.020338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:35.020305    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71c670c-8115-4339-9481-7ac72c34920f" path="/var/lib/kubelet/pods/e71c670c-8115-4339-9481-7ac72c34920f/volumes"
Oct 30 22:19:38.040532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:38.040502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:39.557399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:39.557364    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:39.716600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:39.716569    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:39.921679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:39.921643    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:41.879602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:41.879555    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:42.624436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:42.624400    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:42.783507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:42.782811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:43.284881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:43.284841    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:43.942486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:43.942164    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:44.096479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:44.096445    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:47.472969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:47.472932    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:48.658545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:48.658505    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.522578 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:19:50.522542    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:19:50.768667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.768636    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"]
Oct 30 22:19:50.769155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769141    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" containerName="revoke-keys"
Oct 30 22:19:50.769203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769156    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" containerName="revoke-keys"
Oct 30 22:19:50.769203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769194    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy"
Oct 30 22:19:50.769203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769199    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769206    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" containerName="payload-processing"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769211    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" containerName="payload-processing"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769220    4410 cpu_manager.go:414] "containerMap: removing container" podUID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" containerName="payload-pre-processing"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769226    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" containerName="payload-pre-processing"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769295    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1325f13b-93df-4e61-80fc-4ca8f59e9c5a" containerName="payload-processing"
Oct 30 22:19:50.769302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769301    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71c670c-8115-4339-9481-7ac72c34920f" containerName="istio-proxy"
Oct 30 22:19:50.769482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769309    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f74f11-5b83-44fa-b1a3-9960a81b4cb0" containerName="payload-pre-processing"
Oct 30 22:19:50.769482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.769319    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cdd32e-04e6-4dfb-aa56-d8e9c11df047" containerName="revoke-keys"
Oct 30 22:19:50.774050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.774002    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.834895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.834857    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.835070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.834953    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxlv\" (UniqueName: \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.835070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.834995    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.875708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.875668    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:50.875708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.875679    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:50.875708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.875709    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:50.875990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.875749    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:19:50.879063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.879040    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"]
Oct 30 22:19:50.891328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.891308    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.946678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.946641    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.947041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.947005    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxlv\" (UniqueName: \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.947192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.947072    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.947915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.947883    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:50.950540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:50.950516    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:51.004790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:51.004738    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxlv\" (UniqueName: \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\") pod \"maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:51.086911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:51.086825    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:51.467719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:51.467694    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"]
Oct 30 22:19:51.469532 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:19:51.469489    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f55233_71e2_4a4f_a719_25ec115b4bec.slice/crio-b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9 WatchSource:0}: Error finding container b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9: Status 404 returned error can't find the container with id b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9
Oct 30 22:19:51.889749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:51.889719    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw" event={"ID":"28f55233-71e2-4a4f-a719-25ec115b4bec","Type":"ContainerStarted","Data":"d469d985cccc32ca22645eb3ac9e61a7f3afcb7f62652ec2aa9bb69a1a8607ca"}
Oct 30 22:19:51.890174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:51.889768    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw" event={"ID":"28f55233-71e2-4a4f-a719-25ec115b4bec","Type":"ContainerStarted","Data":"b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9"}
Oct 30 22:19:52.050791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:52.050672    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw" podStartSLOduration=2.050659026 podStartE2EDuration="2.050659026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:52.003684207 +0000 UTC m=+2245.092422024" watchObservedRunningTime="2026-10-30 22:19:52.050659026 +0000 UTC m=+2245.139396825"
Oct 30 22:19:52.896325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:52.896294    4410 generic.go:356] "Generic (PLEG): container finished" podID="28f55233-71e2-4a4f-a719-25ec115b4bec" containerID="d469d985cccc32ca22645eb3ac9e61a7f3afcb7f62652ec2aa9bb69a1a8607ca" exitCode=0
Oct 30 22:19:52.896842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:52.896350    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw" event={"ID":"28f55233-71e2-4a4f-a719-25ec115b4bec","Type":"ContainerDied","Data":"d469d985cccc32ca22645eb3ac9e61a7f3afcb7f62652ec2aa9bb69a1a8607ca"}
Oct 30 22:19:54.051566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.051544    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:54.087440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.087413    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\") pod \"28f55233-71e2-4a4f-a719-25ec115b4bec\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") "
Oct 30 22:19:54.087575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.087513    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\" (UniqueName: \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\") pod \"28f55233-71e2-4a4f-a719-25ec115b4bec\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") "
Oct 30 22:19:54.087575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.087568    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\") pod \"28f55233-71e2-4a4f-a719-25ec115b4bec\" (UID: \"28f55233-71e2-4a4f-a719-25ec115b4bec\") "
Oct 30 22:19:54.087999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.087973    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca" pod "28f55233-71e2-4a4f-a719-25ec115b4bec" (UID: "28f55233-71e2-4a4f-a719-25ec115b4bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:54.088229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.088214    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:54.089435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.089411    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca" pod "28f55233-71e2-4a4f-a719-25ec115b4bec" (UID: "28f55233-71e2-4a4f-a719-25ec115b4bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:54.089630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.089615    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv" pod "28f55233-71e2-4a4f-a719-25ec115b4bec" (UID: "28f55233-71e2-4a4f-a719-25ec115b4bec"). InnerVolumeSpecName "kube-api-access-rtxlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:54.190204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.190129    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f55233-71e2-4a4f-a719-25ec115b4bec-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:54.190204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.190170    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxlv\" (UniqueName: \"kubernetes.io/projected/28f55233-71e2-4a4f-a719-25ec115b4bec-kube-api-access-rtxlv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:54.908565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.908529    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw" event={"ID":"28f55233-71e2-4a4f-a719-25ec115b4bec","Type":"ContainerDied","Data":"b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9"}
Oct 30 22:19:54.908565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.908546    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"
Oct 30 22:19:54.908794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:54.908612    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17387bb897dc99ada6f14e90423b9fa140ddf6af872b1f123e511681e23c5e9"
Oct 30 22:19:55.416198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:55.416163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.575572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:55.575536    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.779358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:55.779284    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:58.687358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:58.687312    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.340856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:59.340802    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.697262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:19:59.697223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:00.670295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:00.670255    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:07.235304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:07.235269    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:12.219539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:12.219496    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.129706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:13.129672    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.326645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:13.326608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.507618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:13.507588    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.445276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:16.445192    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.616925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:16.616892    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:17.600185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:17.600139    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:20.522828 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:20:20.522753    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:20:21.059924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.059454    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"]
Oct 30 22:20:21.108529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.108503    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09489e22-98784"]
Oct 30 22:20:21.117937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.117915    4410 generic.go:356] "Generic (PLEG): container finished" podID="ba18dc36-4920-49b2-a111-498e08aa3344" containerID="06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335" exitCode=0
Oct 30 22:20:21.118049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.117989    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerDied","Data":"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"}
Oct 30 22:20:21.118364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.118351    4410 scope.go:122] "RemoveContainer" containerID="06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"
Oct 30 22:20:21.119897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.119859    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:21.119972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.119939    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:21.120010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:21.119972    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:22.125587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:22.125550    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerStarted","Data":"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"}
Oct 30 22:20:22.225739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:22.225706    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:20:22.225992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:22.225971    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:20:22.226044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:22.226017    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:20:22.531307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:22.530138    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:23.019310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:23.019275    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b" path="/var/lib/kubelet/pods/aa82f573-3bc0-4f8d-9b3d-699e0f6e9c0b/volumes"
Oct 30 22:20:23.225656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:23.225624    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:20:23.226072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:23.225670    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:20:24.225724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:24.225687    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:20:24.226174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:24.225744    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:20:25.226053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:25.226016    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.226445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:25.226092    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:20:26.226287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:26.226252    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.226677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:26.226316    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:20:26.538054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:26.537969    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.230383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:27.230350    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:20:27.230753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:27.230599    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:20:27.231195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:27.231173    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:20:27.529872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:27.529377    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:28.294429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:28.294390    4410 scope.go:122] "RemoveContainer" containerID="9fdb7aca99622f0354f17b26362748c2970c791e11f8c7eb61404c92d82a8e8c"
Oct 30 22:20:31.176283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:31.176241    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:31.714804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:31.714729    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:34.630949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:34.630912    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.015223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:41.015122    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:46.127060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:46.127025    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:49.723462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:49.723417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:50.225146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:50.225112    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:50.522437 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:20:50.522353    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:20:52.894815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:52.894758    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:57.747044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:57.747009    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:59.943212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.943170    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"]
Oct 30 22:20:59.943875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.943853    4410 cpu_manager.go:414] "containerMap: removing container" podUID="28f55233-71e2-4a4f-a719-25ec115b4bec" containerName="revoke-keys"
Oct 30 22:20:59.943998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.943876    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f55233-71e2-4a4f-a719-25ec115b4bec" containerName="revoke-keys"
Oct 30 22:20:59.944060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.944048    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f55233-71e2-4a4f-a719-25ec115b4bec" containerName="revoke-keys"
Oct 30 22:20:59.946928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.946905    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:20:59.999990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:20:59.999937    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:00.002310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.002292    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:00.041830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.041800    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.041997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.041874    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcz4r\" (UniqueName: \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.042066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.042004    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.049757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.049728    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:21:00.049889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.049823    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:00.052726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.052705    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"]
Oct 30 22:21:00.143790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.143748    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.143947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.143928    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.144029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.144006    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcz4r\" (UniqueName: \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.144447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.144431    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.146031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.146013    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.201318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.201269    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcz4r\" (UniqueName: \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\") pod \"maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.259888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.259865    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:00.631143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:00.631108    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"]
Oct 30 22:21:00.632274 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:21:00.632247    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode518e104_3767_4fde_8b51_a07a7e8a3cc2.slice/crio-8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49 WatchSource:0}: Error finding container 8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49: Status 404 returned error can't find the container with id 8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49
Oct 30 22:21:01.372244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:01.372168    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v" event={"ID":"e518e104-3767-4fde-8b51-a07a7e8a3cc2","Type":"ContainerStarted","Data":"08cc3f592b889f8bfe0a3e692533e1977dfa97bdbc8174072d7ff42154876b5f"}
Oct 30 22:21:01.372244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:01.372229    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v" event={"ID":"e518e104-3767-4fde-8b51-a07a7e8a3cc2","Type":"ContainerStarted","Data":"8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49"}
Oct 30 22:21:01.488528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:01.488473    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v" podStartSLOduration=2.488458363 podStartE2EDuration="2.488458363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:01.486842035 +0000 UTC m=+2314.575579837" watchObservedRunningTime="2026-10-30 22:21:01.488458363 +0000 UTC m=+2314.577196167"
Oct 30 22:21:02.162831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:02.162793    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.380134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:02.380094    4410 generic.go:356] "Generic (PLEG): container finished" podID="e518e104-3767-4fde-8b51-a07a7e8a3cc2" containerID="08cc3f592b889f8bfe0a3e692533e1977dfa97bdbc8174072d7ff42154876b5f" exitCode=0
Oct 30 22:21:02.380587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:02.380160    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v" event={"ID":"e518e104-3767-4fde-8b51-a07a7e8a3cc2","Type":"ContainerDied","Data":"08cc3f592b889f8bfe0a3e692533e1977dfa97bdbc8174072d7ff42154876b5f"}
Oct 30 22:21:02.715589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:02.715551    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.883381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:02.883348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:03.523897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.523874    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:03.587891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.587862    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\" (UniqueName: \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\") pod \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") "
Oct 30 22:21:03.588056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.587973    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\") pod \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") "
Oct 30 22:21:03.588056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.588012    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\") pod \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\" (UID: \"e518e104-3767-4fde-8b51-a07a7e8a3cc2\") "
Oct 30 22:21:03.588404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.588381    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca" pod "e518e104-3767-4fde-8b51-a07a7e8a3cc2" (UID: "e518e104-3767-4fde-8b51-a07a7e8a3cc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:03.589998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.589975    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r" pod "e518e104-3767-4fde-8b51-a07a7e8a3cc2" (UID: "e518e104-3767-4fde-8b51-a07a7e8a3cc2"). InnerVolumeSpecName "kube-api-access-bcz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:03.590090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.590059    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca" pod "e518e104-3767-4fde-8b51-a07a7e8a3cc2" (UID: "e518e104-3767-4fde-8b51-a07a7e8a3cc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:03.689597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.689569    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.689597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.689596    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcz4r\" (UniqueName: \"kubernetes.io/projected/e518e104-3767-4fde-8b51-a07a7e8a3cc2-kube-api-access-bcz4r\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.689815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.689611    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e518e104-3767-4fde-8b51-a07a7e8a3cc2-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.868631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:03.868597    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.166837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.166807    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.332987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.332949    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.348564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.348528    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"]
Oct 30 22:21:04.352658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.352630    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-577ea8-7c7lw"]
Oct 30 22:21:04.393765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.393741    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"
Oct 30 22:21:04.393915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.393806    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v" event={"ID":"e518e104-3767-4fde-8b51-a07a7e8a3cc2","Type":"ContainerDied","Data":"8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49"}
Oct 30 22:21:04.393915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:04.393847    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee5d33ceb4b6a90f0a8b31d3764c3d1793cb53f5ae0ae960414ccf48115dd49"
Oct 30 22:21:05.019097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:05.019058    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f55233-71e2-4a4f-a719-25ec115b4bec" path="/var/lib/kubelet/pods/28f55233-71e2-4a4f-a719-25ec115b4bec/volumes"
Oct 30 22:21:09.443490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:09.443451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.693052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:12.693017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.513695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:13.513661    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:21:13.563843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:13.563811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.627927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:13.627854    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:21:13.978365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:13.978331    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.205067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:14.205035    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.390030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:14.389991    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.569143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:14.569106    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.769525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:14.769435    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.281160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:16.281128    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:18.514096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.514057    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" podUID="b750bfe7-3374-4051-9f07-d585b615eebb" containerName="payload-processing" containerID="cri-o://9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c" gracePeriod=25
Oct 30 22:21:18.629206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.629150    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" podUID="86ac0830-d68d-4a76-9d6e-401e274169df" containerName="payload-pre-processing" containerID="cri-o://aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452" gracePeriod=25
Oct 30 22:21:18.783671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.783610    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:21:18.862102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.862068    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\") pod \"b750bfe7-3374-4051-9f07-d585b615eebb\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") "
Oct 30 22:21:18.862357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.862336    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\" (UniqueName: \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\") pod \"b750bfe7-3374-4051-9f07-d585b615eebb\" (UID: \"b750bfe7-3374-4051-9f07-d585b615eebb\") "
Oct 30 22:21:18.862521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.862491    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume" pod "b750bfe7-3374-4051-9f07-d585b615eebb" (UID: "b750bfe7-3374-4051-9f07-d585b615eebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:18.862921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.862897    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b750bfe7-3374-4051-9f07-d585b615eebb-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.864393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.864367    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx" pod "b750bfe7-3374-4051-9f07-d585b615eebb" (UID: "b750bfe7-3374-4051-9f07-d585b615eebb"). InnerVolumeSpecName "kube-api-access-6smtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:18.886841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.886812    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:21:18.963931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.963901    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\") pod \"86ac0830-d68d-4a76-9d6e-401e274169df\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") "
Oct 30 22:21:18.964095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.964006    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\" (UniqueName: \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\") pod \"86ac0830-d68d-4a76-9d6e-401e274169df\" (UID: \"86ac0830-d68d-4a76-9d6e-401e274169df\") "
Oct 30 22:21:18.964298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.964275    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume" pod "86ac0830-d68d-4a76-9d6e-401e274169df" (UID: "86ac0830-d68d-4a76-9d6e-401e274169df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:18.964447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.964431    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac0830-d68d-4a76-9d6e-401e274169df-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.964508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.964452    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smtx\" (UniqueName: \"kubernetes.io/projected/b750bfe7-3374-4051-9f07-d585b615eebb-kube-api-access-6smtx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.965976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:18.965957    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58" pod "86ac0830-d68d-4a76-9d6e-401e274169df" (UID: "86ac0830-d68d-4a76-9d6e-401e274169df"). InnerVolumeSpecName "kube-api-access-krv58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:19.065439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.065364    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krv58\" (UniqueName: \"kubernetes.io/projected/86ac0830-d68d-4a76-9d6e-401e274169df-kube-api-access-krv58\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:19.480962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.480917    4410 generic.go:356] "Generic (PLEG): container finished" podID="b750bfe7-3374-4051-9f07-d585b615eebb" containerID="9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c" exitCode=0
Oct 30 22:21:19.481176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.480988    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"
Oct 30 22:21:19.481176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.481001    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" event={"ID":"b750bfe7-3374-4051-9f07-d585b615eebb","Type":"ContainerDied","Data":"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"}
Oct 30 22:21:19.481176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.481092    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd" event={"ID":"b750bfe7-3374-4051-9f07-d585b615eebb","Type":"ContainerDied","Data":"d166d859c1c3fdde6a5a8ca0c0b5b3fd06869ce6f84384227ef5fdde9fcf1fba"}
Oct 30 22:21:19.481176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.481129    4410 scope.go:122] "RemoveContainer" containerID="9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"
Oct 30 22:21:19.482477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.482450    4410 generic.go:356] "Generic (PLEG): container finished" podID="86ac0830-d68d-4a76-9d6e-401e274169df" containerID="aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452" exitCode=0
Oct 30 22:21:19.483009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.482507    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"
Oct 30 22:21:19.483009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.482542    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" event={"ID":"86ac0830-d68d-4a76-9d6e-401e274169df","Type":"ContainerDied","Data":"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"}
Oct 30 22:21:19.483009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.482588    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w" event={"ID":"86ac0830-d68d-4a76-9d6e-401e274169df","Type":"ContainerDied","Data":"5ec4d6cf76925d11db5942d9e70b5d55f5918e8f09f49f722a3b9478f6fe9920"}
Oct 30 22:21:19.495698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.495562    4410 scope.go:122] "RemoveContainer" containerID="9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"
Oct 30 22:21:19.496232 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:19.496204    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c\": container with ID starting with 9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c not found: ID does not exist" containerID="9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"
Oct 30 22:21:19.496325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.496241    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c"} err="failed to get container status \"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c\": rpc error: code = NotFound desc = could not find container \"9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c\": container with ID starting with 9d731621750aa76b28d61382aaef0a427cac1dbd28abb92a5b26ce5a7209825c not found: ID does not exist"
Oct 30 22:21:19.496325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.496265    4410 scope.go:122] "RemoveContainer" containerID="aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"
Oct 30 22:21:19.504925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.504902    4410 scope.go:122] "RemoveContainer" containerID="aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"
Oct 30 22:21:19.505181 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:19.505161    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452\": container with ID starting with aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452 not found: ID does not exist" containerID="aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"
Oct 30 22:21:19.505241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.505188    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452"} err="failed to get container status \"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452\": rpc error: code = NotFound desc = could not find container \"aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452\": container with ID starting with aa61eaa37f9de77094323209c7f8dcb07d1871e9134862c817001b5b07988452 not found: ID does not exist"
Oct 30 22:21:19.646636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.646600    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:21:19.695513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.695483    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-569f4b4d6c-lgn2w"]
Oct 30 22:21:19.803816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.803733    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:21:19.807672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:19.807647    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-86db66d579-k9lpd"]
Oct 30 22:21:20.522657 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:20.522623    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:21:20.840006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:20.839922    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:20.998595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:20.998563    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:21.020827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:21.020798    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ac0830-d68d-4a76-9d6e-401e274169df" path="/var/lib/kubelet/pods/86ac0830-d68d-4a76-9d6e-401e274169df/volumes"
Oct 30 22:21:21.021283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:21.021265    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b750bfe7-3374-4051-9f07-d585b615eebb" path="/var/lib/kubelet/pods/b750bfe7-3374-4051-9f07-d585b615eebb/volumes"
Oct 30 22:21:21.169170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:21.169134    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:21.320790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:21.320739    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:21.472534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:21.472450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.153255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:26.153221    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.884638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:26.884605    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:28.039412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:28.039379    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:28.320437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:28.320355    4410 scope.go:122] "RemoveContainer" containerID="12dad544a364964b2b3d15a8840c01e713eba2b52166c6c09a1b6c9ab0ad2dac"
Oct 30 22:21:28.329359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:28.329335    4410 scope.go:122] "RemoveContainer" containerID="c1b05809ca96f07030816cdd3b3836b1043b9c53b62f7829fe7e296869d6d606"
Oct 30 22:21:28.775749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:28.775694    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:28.947348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:28.947313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:29.119142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:29.119067    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:31.152675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:31.152640    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:21:31.153178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:31.152949    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy" containerID="cri-o://deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08" gracePeriod=30
Oct 30 22:21:31.180292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:31.180259    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:31.382543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:31.382512    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.063286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.063252    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.226311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.226277    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.432896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.432860    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:21:32.433337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433323    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e518e104-3767-4fde-8b51-a07a7e8a3cc2" containerName="revoke-keys"
Oct 30 22:21:32.433390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433339    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e518e104-3767-4fde-8b51-a07a7e8a3cc2" containerName="revoke-keys"
Oct 30 22:21:32.433390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433374    4410 cpu_manager.go:414] "containerMap: removing container" podUID="86ac0830-d68d-4a76-9d6e-401e274169df" containerName="payload-pre-processing"
Oct 30 22:21:32.433390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433379    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ac0830-d68d-4a76-9d6e-401e274169df" containerName="payload-pre-processing"
Oct 30 22:21:32.433488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433395    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b750bfe7-3374-4051-9f07-d585b615eebb" containerName="payload-processing"
Oct 30 22:21:32.433488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433402    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b750bfe7-3374-4051-9f07-d585b615eebb" containerName="payload-processing"
Oct 30 22:21:32.433488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433478    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b750bfe7-3374-4051-9f07-d585b615eebb" containerName="payload-processing"
Oct 30 22:21:32.433582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433490    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e518e104-3767-4fde-8b51-a07a7e8a3cc2" containerName="revoke-keys"
Oct 30 22:21:32.433582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.433500    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ac0830-d68d-4a76-9d6e-401e274169df" containerName="payload-pre-processing"
Oct 30 22:21:32.438250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.438224    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.495840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.495811    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:21:32.557912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.542044    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-955887\""
Oct 30 22:21:32.557912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.542713    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-955887-dockercfg-tnhqv\""
Oct 30 22:21:32.559796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.559430    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdf2\" (UniqueName: \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.559796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.559556    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.587486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.587446    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:21:32.591665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.591631    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.655957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.655915    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:21:32.660556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.660523    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdf2\" (UniqueName: \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.660724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.660627    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.661695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.661664    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.717922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.717860    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdf2\" (UniqueName: \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\") pod \"payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.751003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.750968    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:32.761832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.761802    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9vp\" (UniqueName: \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.761999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.761971    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.863399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.863367    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9vp\" (UniqueName: \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.863584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.863446    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.864125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.864102    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:32.920494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:32.920462    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9vp\" (UniqueName: \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\") pod \"payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:33.125905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.125844    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:21:33.127464 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:21:33.127432    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed13549_2bb8_430d_8f0c_c10916ea81b6.slice/crio-938f384197660aebcec9b34dcec4a6da1309d561a5783a51b57585dc9ca8e69d WatchSource:0}: Error finding container 938f384197660aebcec9b34dcec4a6da1309d561a5783a51b57585dc9ca8e69d: Status 404 returned error can't find the container with id 938f384197660aebcec9b34dcec4a6da1309d561a5783a51b57585dc9ca8e69d
Oct 30 22:21:33.207518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.207494    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:33.588241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.588191    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" event={"ID":"1ed13549-2bb8-430d-8f0c-c10916ea81b6","Type":"ContainerStarted","Data":"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"}
Oct 30 22:21:33.588623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.588266    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" event={"ID":"1ed13549-2bb8-430d-8f0c-c10916ea81b6","Type":"ContainerStarted","Data":"938f384197660aebcec9b34dcec4a6da1309d561a5783a51b57585dc9ca8e69d"}
Oct 30 22:21:33.588717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.588695    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:33.590871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.590844    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:21:33.592330 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:21:33.592303    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c15231_da63_4c54_af02_a15a52569ad0.slice/crio-b423d1a3b6919afc467319b1d69c4254564a974e48cd91d1d80eea5590801c00 WatchSource:0}: Error finding container b423d1a3b6919afc467319b1d69c4254564a974e48cd91d1d80eea5590801c00: Status 404 returned error can't find the container with id b423d1a3b6919afc467319b1d69c4254564a974e48cd91d1d80eea5590801c00
Oct 30 22:21:33.717981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:33.717914    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" podStartSLOduration=1.717896304 podStartE2EDuration="1.717896304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:33.715185577 +0000 UTC m=+2346.803923380" watchObservedRunningTime="2026-10-30 22:21:33.717896304 +0000 UTC m=+2346.806634107"
Oct 30 22:21:34.595272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:34.595233    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" event={"ID":"c3c15231-da63-4c54-af02-a15a52569ad0","Type":"ContainerStarted","Data":"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"}
Oct 30 22:21:34.595666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:34.595297    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" event={"ID":"c3c15231-da63-4c54-af02-a15a52569ad0","Type":"ContainerStarted","Data":"b423d1a3b6919afc467319b1d69c4254564a974e48cd91d1d80eea5590801c00"}
Oct 30 22:21:34.595666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:34.595399    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:34.707226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:34.707161    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" podStartSLOduration=2.7071430359999997 podStartE2EDuration="2.707143036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:34.70613694 +0000 UTC m=+2347.794874752" watchObservedRunningTime="2026-10-30 22:21:34.707143036 +0000 UTC m=+2347.795880840"
Oct 30 22:21:36.406013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.405989    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:21:36.507191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507124    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507170    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507212    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507255    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507286    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507412    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507528    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507541    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:36.507813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507550    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:36.507984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507743    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:36.507984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507755    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:36.507984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507819    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.507984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.507858    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\") pod \"56a326d1-cf19-4060-97bc-be42cd1426ca\" (UID: \"56a326d1-cf19-4060-97bc-be42cd1426ca\") "
Oct 30 22:21:36.508266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508240    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:36.508372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508344    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.508372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508361    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.508372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508370    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a326d1-cf19-4060-97bc-be42cd1426ca-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.508540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508379    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.508540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.508388    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.509549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.509526    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:36.509621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.509571    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:36.509741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.509728    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "kube-api-access-gpzx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:36.509967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.509944    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo" pod "56a326d1-cf19-4060-97bc-be42cd1426ca" (UID: "56a326d1-cf19-4060-97bc-be42cd1426ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:36.607736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.607707    4410 generic.go:356] "Generic (PLEG): container finished" podID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerID="deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08" exitCode=0
Oct 30 22:21:36.607879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.607755    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"
Oct 30 22:21:36.607879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.607810    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerDied","Data":"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"}
Oct 30 22:21:36.607962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.607907    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s" event={"ID":"56a326d1-cf19-4060-97bc-be42cd1426ca","Type":"ContainerDied","Data":"9f71895b6c6deb0c667a9effc604e308af2b52f7c1f7e75f8b1f042fdd855a48"}
Oct 30 22:21:36.607962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.607941    4410 scope.go:122] "RemoveContainer" containerID="deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"
Oct 30 22:21:36.609242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.609222    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.609334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.609243    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.609334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.609254    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzx4\" (UniqueName: \"kubernetes.io/projected/56a326d1-cf19-4060-97bc-be42cd1426ca-kube-api-access-gpzx4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.609334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.609264    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a326d1-cf19-4060-97bc-be42cd1426ca-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.619559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.619507    4410 scope.go:122] "RemoveContainer" containerID="617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"
Oct 30 22:21:36.631086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.631069    4410 scope.go:122] "RemoveContainer" containerID="deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"
Oct 30 22:21:36.631317 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:36.631299    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08\": container with ID starting with deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08 not found: ID does not exist" containerID="deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"
Oct 30 22:21:36.631386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.631326    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08"} err="failed to get container status \"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08\": rpc error: code = NotFound desc = could not find container \"deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08\": container with ID starting with deb7afa4bf45ce41afbece1fb98929c999e4516af1a2a4c27219eac247a67d08 not found: ID does not exist"
Oct 30 22:21:36.631386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.631347    4410 scope.go:122] "RemoveContainer" containerID="617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"
Oct 30 22:21:36.631599 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:36.631584    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a\": container with ID starting with 617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a not found: ID does not exist" containerID="617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"
Oct 30 22:21:36.631654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.631602    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a"} err="failed to get container status \"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a\": rpc error: code = NotFound desc = could not find container \"617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a\": container with ID starting with 617a50fcebc960f7f9886475a9ebae8ad49b430b88f6484edac41eed945fc66a not found: ID does not exist"
Oct 30 22:21:36.774699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.774612    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:21:36.777621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:36.777597    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-577ea8-openshift-default-654dd86f59-s688s"]
Oct 30 22:21:37.019803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:37.019758    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" path="/var/lib/kubelet/pods/56a326d1-cf19-4060-97bc-be42cd1426ca/volumes"
Oct 30 22:21:37.029492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:37.029425    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:42.508484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:42.508445    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:44.598975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:44.598946    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:21:45.603958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:45.603927    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:21:49.446827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:49.446792    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:50.522464 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:21:50.522429    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:21:52.994518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:52.994480    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:53.162184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:53.162128    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:53.902199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:53.902165    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:54.404972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.404938    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:21:54.405422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405408    4410 cpu_manager.go:414] "containerMap: removing container" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.405422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405422    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.405559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405445    4410 cpu_manager.go:414] "containerMap: removing container" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.405559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405451    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.405559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405523    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.405559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.405537    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a326d1-cf19-4060-97bc-be42cd1426ca" containerName="istio-proxy"
Oct 30 22:21:54.408691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.408672    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.503285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.503255    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-086575-openshift-default-dockercfg-ql5ww\""
Oct 30 22:21:54.508710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.508682    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.508954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.508933    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.509103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.509084    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.509335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.509317    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.509498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.509481    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.509661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.509639    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.509903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.509872    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.510278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.510257    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmjl\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.510544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.510516    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.546819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.542985    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:54.555555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.555515    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:21:54.612173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612124    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612179    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612209    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612281    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612332    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612365    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612415    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612481    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmjl\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612558    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.612927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.612692    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.613330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.613307    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.613815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.613663    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.614119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.614084    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.614230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.614128    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.615974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.615950    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.618978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.618959    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.669501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.669448    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.669850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.669832    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmjl\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\") pod \"e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") " pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.725117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.724723    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:54.837385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:54.837346    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:55.154973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.154947    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:21:55.156555 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:21:55.156531    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd895c5f9_183f_4a98_bdf2_49aab76ea9af.slice/crio-689d3ccb4ab39a6e9481eb4fc737a82401db31174643f0b3db6e01d37ead5506 WatchSource:0}: Error finding container 689d3ccb4ab39a6e9481eb4fc737a82401db31174643f0b3db6e01d37ead5506: Status 404 returned error can't find the container with id 689d3ccb4ab39a6e9481eb4fc737a82401db31174643f0b3db6e01d37ead5506
Oct 30 22:21:55.159884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.159844    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:55.159982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.159928    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:55.159982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.159971    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:55.729864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.729829    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerStarted","Data":"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"}
Oct 30 22:21:55.730263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.729901    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerStarted","Data":"689d3ccb4ab39a6e9481eb4fc737a82401db31174643f0b3db6e01d37ead5506"}
Oct 30 22:21:55.848989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:55.848939    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podStartSLOduration=1.848926157 podStartE2EDuration="1.848926157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:55.845999406 +0000 UTC m=+2368.934737211" watchObservedRunningTime="2026-10-30 22:21:55.848926157 +0000 UTC m=+2368.937663959"
Oct 30 22:21:56.229841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:56.229800    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:56.725568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:56.725531    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:56.730547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:56.730526    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:56.735826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:56.735806    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:56.736549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:56.736531    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:21:58.701016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:21:58.700981    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.507589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:00.507555    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:01.596597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:01.596541    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:01.955496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:01.955455    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:02.081942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.080163    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"]
Oct 30 22:22:02.123138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.123108    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7779b9d3-2b2th"]
Oct 30 22:22:02.168280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.168237    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:02.356609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.356527    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:22:02.360407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.360391    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.409576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.409548    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52ws\" (UniqueName: \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.409683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.409596    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.413155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.413138    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-086575-dockercfg-gqs8c\""
Oct 30 22:22:02.418824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.418803    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-086575\""
Oct 30 22:22:02.420593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.420573    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:22:02.521827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.510555    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52ws\" (UniqueName: \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.521827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.518901    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.521827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.520020    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.546043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.545999    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:22:02.551042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.551014    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.574290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.574258    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:22:02.575711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.575683    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52ws\" (UniqueName: \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\") pod \"payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.620824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.620754    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.620824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.620804    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb49n\" (UniqueName: \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.674481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.674449    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:02.722652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.722609    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.722850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.722723    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb49n\" (UniqueName: \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.723335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.723309    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.779685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.779653    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb49n\" (UniqueName: \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\") pod \"payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:02.865990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:02.865961    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:03.020369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.020333    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2d953a-7cb9-41ba-83ba-364d2e70b404" path="/var/lib/kubelet/pods/ab2d953a-7cb9-41ba-83ba-364d2e70b404/volumes"
Oct 30 22:22:03.057383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.057359    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:22:03.058437 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:22:03.058403    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ca7788_5a0c_4b1f_b5be_c4a393cf14fe.slice/crio-4e5e16c5907ccb030380e06827e49afd05607a3a2544e0ce372d652ee721882a WatchSource:0}: Error finding container 4e5e16c5907ccb030380e06827e49afd05607a3a2544e0ce372d652ee721882a: Status 404 returned error can't find the container with id 4e5e16c5907ccb030380e06827e49afd05607a3a2544e0ce372d652ee721882a
Oct 30 22:22:03.238811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.238766    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:22:03.240374 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:22:03.240346    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f423c8f_25c4_4402_b660_c0b6adc125a9.slice/crio-14e25bf24ca694bbd13171452d085113c608c47f5e0019b9c8bcb059759cb79b WatchSource:0}: Error finding container 14e25bf24ca694bbd13171452d085113c608c47f5e0019b9c8bcb059759cb79b: Status 404 returned error can't find the container with id 14e25bf24ca694bbd13171452d085113c608c47f5e0019b9c8bcb059759cb79b
Oct 30 22:22:03.777224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.777186    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" event={"ID":"5f423c8f-25c4-4402-b660-c0b6adc125a9","Type":"ContainerStarted","Data":"4551b481fa51dc88ca66b0e02e32fe046a56bbfb948ad3964ea2e0c3a7e9afd2"}
Oct 30 22:22:03.777625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.777244    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" event={"ID":"5f423c8f-25c4-4402-b660-c0b6adc125a9","Type":"ContainerStarted","Data":"14e25bf24ca694bbd13171452d085113c608c47f5e0019b9c8bcb059759cb79b"}
Oct 30 22:22:03.777625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.777270    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:03.778747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.778722    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" event={"ID":"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe","Type":"ContainerStarted","Data":"86ce50f47d9588534b0133800e8ecf46f08b8cad70f7c6ef6f1247e6caafac5c"}
Oct 30 22:22:03.778887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.778768    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" event={"ID":"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe","Type":"ContainerStarted","Data":"4e5e16c5907ccb030380e06827e49afd05607a3a2544e0ce372d652ee721882a"}
Oct 30 22:22:03.778887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.778816    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:03.900091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:03.900041    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" podStartSLOduration=1.9000282849999999 podStartE2EDuration="1.900028285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:03.898624719 +0000 UTC m=+2376.987362523" watchObservedRunningTime="2026-10-30 22:22:03.900028285 +0000 UTC m=+2376.988766088"
Oct 30 22:22:06.574732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:06.574698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.670219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:08.670189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.464730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:13.464640    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:14.788419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:14.788388    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:22:14.789115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:14.789093    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:22:14.903658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:14.903607    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" podStartSLOduration=12.903593729 podStartE2EDuration="12.903593729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:04.035983301 +0000 UTC m=+2377.124721125" watchObservedRunningTime="2026-10-30 22:22:14.903593729 +0000 UTC m=+2387.992331532"
Oct 30 22:22:18.542662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:18.542629    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:19.041029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.040994    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"]
Oct 30 22:22:19.048282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.048253    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.101749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.101719    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"]
Oct 30 22:22:19.147755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.147718    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:19.147932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.147901    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:22:19.148008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.147961    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:19.149326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.149306    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:19.216477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.216433    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hdc\" (UniqueName: \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.216656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.216584    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.216722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.216661    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.317506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.317426    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hdc\" (UniqueName: \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.317657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.317510    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.317657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.317545    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.318344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.318321    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.319968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.319947    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.373503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.373476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hdc\" (UniqueName: \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:19.663372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:19.663337    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:20.036751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:20.036724    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"]
Oct 30 22:22:20.038397 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:22:20.038371    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa678f3_e585_4b2e_9e2e_ef701951b959.slice/crio-9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366 WatchSource:0}: Error finding container 9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366: Status 404 returned error can't find the container with id 9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366
Oct 30 22:22:20.521760 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:22:20.521724    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:22:20.878512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:20.878473    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp" event={"ID":"6aa678f3-e585-4b2e-9e2e-ef701951b959","Type":"ContainerStarted","Data":"dd38a6aba516627ce80470635eeac1f745c6e5fe530d0f02152aaed21cdac78a"}
Oct 30 22:22:20.878922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:20.878533    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp" event={"ID":"6aa678f3-e585-4b2e-9e2e-ef701951b959","Type":"ContainerStarted","Data":"9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366"}
Oct 30 22:22:20.993646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:20.993595    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp" podStartSLOduration=2.993581931 podStartE2EDuration="2.993581931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:20.991192341 +0000 UTC m=+2394.079930155" watchObservedRunningTime="2026-10-30 22:22:20.993581931 +0000 UTC m=+2394.082319734"
Oct 30 22:22:21.886410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:21.886327    4410 generic.go:356] "Generic (PLEG): container finished" podID="6aa678f3-e585-4b2e-9e2e-ef701951b959" containerID="dd38a6aba516627ce80470635eeac1f745c6e5fe530d0f02152aaed21cdac78a" exitCode=0
Oct 30 22:22:21.886410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:21.886370    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp" event={"ID":"6aa678f3-e585-4b2e-9e2e-ef701951b959","Type":"ContainerDied","Data":"dd38a6aba516627ce80470635eeac1f745c6e5fe530d0f02152aaed21cdac78a"}
Oct 30 22:22:22.325361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:22.325332    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:22.923418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:22.923385    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:23.060998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.060970    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:23.169092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.169002    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\") pod \"6aa678f3-e585-4b2e-9e2e-ef701951b959\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") "
Oct 30 22:22:23.169254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.169155    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\") pod \"6aa678f3-e585-4b2e-9e2e-ef701951b959\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") "
Oct 30 22:22:23.169254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.169249    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\" (UniqueName: \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\") pod \"6aa678f3-e585-4b2e-9e2e-ef701951b959\" (UID: \"6aa678f3-e585-4b2e-9e2e-ef701951b959\") "
Oct 30 22:22:23.169508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.169479    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca" pod "6aa678f3-e585-4b2e-9e2e-ef701951b959" (UID: "6aa678f3-e585-4b2e-9e2e-ef701951b959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:23.169828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.169801    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.171193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.171167    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca" pod "6aa678f3-e585-4b2e-9e2e-ef701951b959" (UID: "6aa678f3-e585-4b2e-9e2e-ef701951b959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:23.171281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.171245    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc" pod "6aa678f3-e585-4b2e-9e2e-ef701951b959" (UID: "6aa678f3-e585-4b2e-9e2e-ef701951b959"). InnerVolumeSpecName "kube-api-access-q6hdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:23.272312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.272273    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hdc\" (UniqueName: \"kubernetes.io/projected/6aa678f3-e585-4b2e-9e2e-ef701951b959-kube-api-access-q6hdc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.272312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.272314    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa678f3-e585-4b2e-9e2e-ef701951b959-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.899927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.899898    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"
Oct 30 22:22:23.900105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.899942    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp" event={"ID":"6aa678f3-e585-4b2e-9e2e-ef701951b959","Type":"ContainerDied","Data":"9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366"}
Oct 30 22:22:23.900105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:23.899995    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9173b8138b4292b2ce18b79b466f5ea84772903cedce342f73fec8ba891c7366"
Oct 30 22:22:25.242658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:25.242621    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:25.901435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:25.901398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:26.113113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:26.113046    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:26.956310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:26.956277    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:27.232832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:27.232736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:27.398356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:27.398325    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:27.476475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:27.476436    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"]
Oct 30 22:22:27.525879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:27.525809    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-69a1d2-8sf5v"]
Oct 30 22:22:27.915443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:27.915412    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:29.019763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:29.019729    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e518e104-3767-4fde-8b51-a07a7e8a3cc2" path="/var/lib/kubelet/pods/e518e104-3767-4fde-8b51-a07a7e8a3cc2/volumes"
Oct 30 22:22:32.588244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:32.588207    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:34.381460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:34.381424    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:34.543195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:34.543159    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:34.709124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:34.709086    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:35.479135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.479101    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"]
Oct 30 22:22:35.479587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.479573    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6aa678f3-e585-4b2e-9e2e-ef701951b959" containerName="revoke-keys"
Oct 30 22:22:35.479631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.479588    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa678f3-e585-4b2e-9e2e-ef701951b959" containerName="revoke-keys"
Oct 30 22:22:35.479703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.479694    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa678f3-e585-4b2e-9e2e-ef701951b959" containerName="revoke-keys"
Oct 30 22:22:35.484004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.483987    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.536426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.536405    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:35.539289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.539262    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:35.588692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.588666    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:35.588866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.588702    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:22:35.588866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.588753    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"]
Oct 30 22:22:35.615617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.615590    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjc6s\" (UniqueName: \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.615745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.615623    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.615902 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.615876    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.717214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.717187    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjc6s\" (UniqueName: \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.717331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.717218    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.717331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.717325    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.718004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.717983    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.719623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.719599    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.776805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.776718    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjc6s\" (UniqueName: \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:35.797309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:35.797288    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:36.171173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:36.171145    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"]
Oct 30 22:22:36.172992 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:22:36.172965    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce23d008_ecfd_4429_b881_140b8903f4fd.slice/crio-5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811 WatchSource:0}: Error finding container 5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811: Status 404 returned error can't find the container with id 5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811
Oct 30 22:22:36.988350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:36.988315    4410 generic.go:356] "Generic (PLEG): container finished" podID="ce23d008-ecfd-4429-b881-140b8903f4fd" containerID="1feacd6120eb1a0eb00011a69b97b90f2f00eb5a6328d2a221749c38f29c432e" exitCode=0
Oct 30 22:22:36.988990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:36.988951    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj" event={"ID":"ce23d008-ecfd-4429-b881-140b8903f4fd","Type":"ContainerDied","Data":"1feacd6120eb1a0eb00011a69b97b90f2f00eb5a6328d2a221749c38f29c432e"}
Oct 30 22:22:36.989279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:36.989260    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj" event={"ID":"ce23d008-ecfd-4429-b881-140b8903f4fd","Type":"ContainerStarted","Data":"5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811"}
Oct 30 22:22:37.844610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:37.844578    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:38.148715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.148691    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:38.247390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.247351    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\" (UniqueName: \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\") pod \"ce23d008-ecfd-4429-b881-140b8903f4fd\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") "
Oct 30 22:22:38.247578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.247468    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\") pod \"ce23d008-ecfd-4429-b881-140b8903f4fd\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") "
Oct 30 22:22:38.247578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.247564    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\") pod \"ce23d008-ecfd-4429-b881-140b8903f4fd\" (UID: \"ce23d008-ecfd-4429-b881-140b8903f4fd\") "
Oct 30 22:22:38.247933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.247905    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca" pod "ce23d008-ecfd-4429-b881-140b8903f4fd" (UID: "ce23d008-ecfd-4429-b881-140b8903f4fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:38.248101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.248076    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:38.249558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.249534    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s" pod "ce23d008-ecfd-4429-b881-140b8903f4fd" (UID: "ce23d008-ecfd-4429-b881-140b8903f4fd"). InnerVolumeSpecName "kube-api-access-fjc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:38.249645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.249550    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca" pod "ce23d008-ecfd-4429-b881-140b8903f4fd" (UID: "ce23d008-ecfd-4429-b881-140b8903f4fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:38.349419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.349385    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce23d008-ecfd-4429-b881-140b8903f4fd-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:38.349419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:38.349415    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjc6s\" (UniqueName: \"kubernetes.io/projected/ce23d008-ecfd-4429-b881-140b8903f4fd-kube-api-access-fjc6s\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:39.003170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:39.003143    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"
Oct 30 22:22:39.003378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:39.003144    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj" event={"ID":"ce23d008-ecfd-4429-b881-140b8903f4fd","Type":"ContainerDied","Data":"5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811"}
Oct 30 22:22:39.003378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:39.003279    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb04cd8b0c95756d87fe2162dd1b78193efebf1411aa3c184b1d11173439811"
Oct 30 22:22:40.830250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:40.830215    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:43.017341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:43.017305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:43.201001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:43.200962    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:48.124218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:48.124170    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:50.522321 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:22:50.522282    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:22:52.731005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:52.730972    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:53.025080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:53.024945    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.274064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:57.274009    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.842174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:57.842123    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:58.054137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.054082    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:58.501306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.501270    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"]
Oct 30 22:22:58.501791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.501756    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ce23d008-ecfd-4429-b881-140b8903f4fd" containerName="revoke-keys"
Oct 30 22:22:58.501888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.501804    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce23d008-ecfd-4429-b881-140b8903f4fd" containerName="revoke-keys"
Oct 30 22:22:58.502012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.501996    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce23d008-ecfd-4429-b881-140b8903f4fd" containerName="revoke-keys"
Oct 30 22:22:58.510250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.506559    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.563033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.562997    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:58.565416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.565384    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"]
Oct 30 22:22:58.587926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.587898    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzd4\" (UniqueName: \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.588059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.588033    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.588187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.588171    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.590196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.590175    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:22:58.590286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.590187    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:58.611140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.611118    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:58.689728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.689701    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.689894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.689851    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzd4\" (UniqueName: \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.689968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.689929    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.690346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.690326    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.692210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.692182    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.746363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.746334    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzd4\" (UniqueName: \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\") pod \"maas-api-revoke-sub-e9022fad-brg6k\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.823532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.823479    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:22:58.865582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:58.865549    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:59.199353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.199318    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:59.200927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.200899    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"]
Oct 30 22:22:59.202217 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:22:59.202191    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc7224_d1b3_4bfd_a735_3e89a8e47147.slice/crio-5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b WatchSource:0}: Error finding container 5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b: Status 404 returned error can't find the container with id 5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b
Oct 30 22:22:59.408492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.408460    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:59.410093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.410067    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"]
Oct 30 22:22:59.412448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.412427    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a28730-99mbj"]
Oct 30 22:22:59.747629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:22:59.747536    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:00.124195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:00.124160    4410 generic.go:356] "Generic (PLEG): container finished" podID="9fbc7224-d1b3-4bfd-a735-3e89a8e47147" containerID="2aa714a137bad41a4d780cfed340625f308042087c61e7564ad0619f148d9268" exitCode=0
Oct 30 22:23:00.124358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:00.124241    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k" event={"ID":"9fbc7224-d1b3-4bfd-a735-3e89a8e47147","Type":"ContainerDied","Data":"2aa714a137bad41a4d780cfed340625f308042087c61e7564ad0619f148d9268"}
Oct 30 22:23:00.124358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:00.124288    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k" event={"ID":"9fbc7224-d1b3-4bfd-a735-3e89a8e47147","Type":"ContainerStarted","Data":"5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b"}
Oct 30 22:23:01.021512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.021478    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce23d008-ecfd-4429-b881-140b8903f4fd" path="/var/lib/kubelet/pods/ce23d008-ecfd-4429-b881-140b8903f4fd/volumes"
Oct 30 22:23:01.271009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.270985    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:23:01.321878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.321824    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\") pod \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") "
Oct 30 22:23:01.321990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.321879    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\") pod \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") "
Oct 30 22:23:01.322045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.322025    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\" (UniqueName: \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\") pod \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\" (UID: \"9fbc7224-d1b3-4bfd-a735-3e89a8e47147\") "
Oct 30 22:23:01.322175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.322140    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca" pod "9fbc7224-d1b3-4bfd-a735-3e89a8e47147" (UID: "9fbc7224-d1b3-4bfd-a735-3e89a8e47147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:01.322353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.322337    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:01.323908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.323884    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca" pod "9fbc7224-d1b3-4bfd-a735-3e89a8e47147" (UID: "9fbc7224-d1b3-4bfd-a735-3e89a8e47147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:01.324227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.324210    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4" pod "9fbc7224-d1b3-4bfd-a735-3e89a8e47147" (UID: "9fbc7224-d1b3-4bfd-a735-3e89a8e47147"). InnerVolumeSpecName "kube-api-access-lqzd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:01.423348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.423319    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqzd4\" (UniqueName: \"kubernetes.io/projected/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-kube-api-access-lqzd4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:01.423348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:01.423346    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbc7224-d1b3-4bfd-a735-3e89a8e47147-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.138068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:02.138042    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"
Oct 30 22:23:02.138466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:02.138031    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k" event={"ID":"9fbc7224-d1b3-4bfd-a735-3e89a8e47147","Type":"ContainerDied","Data":"5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b"}
Oct 30 22:23:02.138466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:02.138169    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dbb56a896013cb47a3dd9f4dc414c263c4e44b90b60ce5c5db75d1f9ffbca1b"
Oct 30 22:23:04.282507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:04.282472    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:04.544013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:04.543910    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:04.776598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:04.776565    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.488288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.488252    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:23:05.488692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.488453    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy" containerID="cri-o://696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11" gracePeriod=30
Oct 30 22:23:05.538111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.538078    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 22:23:05.538309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.538288    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" containerID="cri-o://04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272" gracePeriod=30
Oct 30 22:23:05.540912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.540879    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 22:23:05.541123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.541095    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy" containerID="cri-o://f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef" gracePeriod=30
Oct 30 22:23:05.587093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.587065    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:23:05.587283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.587263    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy" containerID="cri-o://ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf" gracePeriod=30
Oct 30 22:23:05.647031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.646983    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"]
Oct 30 22:23:05.647571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.647550    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9fbc7224-d1b3-4bfd-a735-3e89a8e47147" containerName="revoke-keys"
Oct 30 22:23:05.647677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.647573    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbc7224-d1b3-4bfd-a735-3e89a8e47147" containerName="revoke-keys"
Oct 30 22:23:05.647736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.647708    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbc7224-d1b3-4bfd-a735-3e89a8e47147" containerName="revoke-keys"
Oct 30 22:23:05.651363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.651344    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.705272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.705233    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:05.705934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.705908    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:05.706427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.706386    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:05.706522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.706486    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:23:05.708139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.708108    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"]
Oct 30 22:23:05.772763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.772694    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.772954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.772932    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.773020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.772986    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbgp\" (UniqueName: \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.874134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.874105    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbgp\" (UniqueName: \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.874284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.874148    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.874340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.874301    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.874960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.874937    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.876457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.876434    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.934365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.934339    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbgp\" (UniqueName: \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\") pod \"maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:05.967612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:05.967590    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:06.379894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:06.366878    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"]
Oct 30 22:23:07.031669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.031571    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.167683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.167647    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk" event={"ID":"014b60ba-73ac-4802-8414-73984b145cc1","Type":"ContainerStarted","Data":"96c2e5cc834ee2dc4cbd6d1c87a8ee9583b6a90c2d94b3c9a59a119415a5891d"}
Oct 30 22:23:07.167876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.167698    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk" event={"ID":"014b60ba-73ac-4802-8414-73984b145cc1","Type":"ContainerStarted","Data":"5baa552a76f8e3f7e71e8007875786693b141938dd543ce92acf0530688db717"}
Oct 30 22:23:07.201769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.201738    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.282043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.281956    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk" podStartSLOduration=2.281940524 podStartE2EDuration="2.281940524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.280289639 +0000 UTC m=+2440.369027443" watchObservedRunningTime="2026-10-30 22:23:07.281940524 +0000 UTC m=+2440.370678333"
Oct 30 22:23:07.398746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.398704    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.930235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.930201    4410 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fjr78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:07.930426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:07.930258    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:08.212862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:08.211765    4410 generic.go:356] "Generic (PLEG): container finished" podID="014b60ba-73ac-4802-8414-73984b145cc1" containerID="96c2e5cc834ee2dc4cbd6d1c87a8ee9583b6a90c2d94b3c9a59a119415a5891d" exitCode=0
Oct 30 22:23:08.212862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:08.211950    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk" event={"ID":"014b60ba-73ac-4802-8414-73984b145cc1","Type":"ContainerDied","Data":"96c2e5cc834ee2dc4cbd6d1c87a8ee9583b6a90c2d94b3c9a59a119415a5891d"}
Oct 30 22:23:08.814895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:08.814860    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:09.405529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.405430    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:09.538373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.538348    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:09.627542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.627511    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:09.639594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.639568    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\" (UniqueName: \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\") pod \"014b60ba-73ac-4802-8414-73984b145cc1\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") "
Oct 30 22:23:09.639747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.639617    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\") pod \"014b60ba-73ac-4802-8414-73984b145cc1\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") "
Oct 30 22:23:09.639747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.639653    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\") pod \"014b60ba-73ac-4802-8414-73984b145cc1\" (UID: \"014b60ba-73ac-4802-8414-73984b145cc1\") "
Oct 30 22:23:09.640169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.640130    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca" pod "014b60ba-73ac-4802-8414-73984b145cc1" (UID: "014b60ba-73ac-4802-8414-73984b145cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:09.641972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.641939    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp" pod "014b60ba-73ac-4802-8414-73984b145cc1" (UID: "014b60ba-73ac-4802-8414-73984b145cc1"). InnerVolumeSpecName "kube-api-access-8fbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:09.642072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.641967    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca" pod "014b60ba-73ac-4802-8414-73984b145cc1" (UID: "014b60ba-73ac-4802-8414-73984b145cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:09.741148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.741109    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbgp\" (UniqueName: \"kubernetes.io/projected/014b60ba-73ac-4802-8414-73984b145cc1-kube-api-access-8fbgp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.741148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.741150    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.741343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.741166    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b60ba-73ac-4802-8414-73984b145cc1-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.991127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:09.991095    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.226125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.226091    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk" event={"ID":"014b60ba-73ac-4802-8414-73984b145cc1","Type":"ContainerDied","Data":"5baa552a76f8e3f7e71e8007875786693b141938dd543ce92acf0530688db717"}
Oct 30 22:23:10.226342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.226143    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5baa552a76f8e3f7e71e8007875786693b141938dd543ce92acf0530688db717"
Oct 30 22:23:10.226342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.226099    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"
Oct 30 22:23:10.813621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.813591    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:23:10.868983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.868955    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 22:23:10.945091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.945065    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 22:23:10.948337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.948315    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:23:10.957826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.957767    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.957962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.957863    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.957962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.957885    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.957962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.957930    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958053    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958132    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.958173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958150    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.958281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958172    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958211    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958243    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958264    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.958438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958295    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958331    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958404    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.958438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958418    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958450    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958482    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958486    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958561    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958613    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\") pod \"cc9fd184-8208-4667-b204-53d2e5e9581b\" (UID: \"cc9fd184-8208-4667-b204-53d2e5e9581b\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958647    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.958684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958677    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.959069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958701    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.959069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958707    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\") pod \"494e40ea-8a23-4189-8191-0035bf7ed04f\" (UID: \"494e40ea-8a23-4189-8191-0035bf7ed04f\") "
Oct 30 22:23:10.959069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.958931    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.959239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959218    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:10.959751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959435    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.959751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959695    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:10.959751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959735    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9fd184-8208-4667-b204-53d2e5e9581b-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959758    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959790    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959808    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959823    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959837    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959852    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.959988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.959867    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.960464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.960430    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "kube-api-access-wwlw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.960660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.960637    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.960743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.960660    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "kube-api-access-dn5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.960920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.960902    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:10.961021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.961005    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.961555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.961539    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:10.961690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.961677    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy" pod "cc9fd184-8208-4667-b204-53d2e5e9581b" (UID: "cc9fd184-8208-4667-b204-53d2e5e9581b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:10.962030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:10.962010    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy" pod "494e40ea-8a23-4189-8191-0035bf7ed04f" (UID: "494e40ea-8a23-4189-8191-0035bf7ed04f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.060440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060414    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.060557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060454    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.060557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060487    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.060557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060518    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.060557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060551    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.060811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060609    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.060811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060694    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.060811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.060728    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.061156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061043    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:11.061242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061127    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.061308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061266    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.061366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.061366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061327    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.061481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061414    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.061481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061447    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.061593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061487    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.061593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061511    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.061593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061551    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.061593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061585    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.061838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061617    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.061838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061677    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.061838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061704    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\") pod \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\" (UID: \"0cbeca0a-5b04-4bbc-b01d-3f693635106a\") "
Oct 30 22:23:11.061838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061744    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\") pod \"52d5355c-3025-47f3-8333-39439647415d\" (UID: \"52d5355c-3025-47f3-8333-39439647415d\") "
Oct 30 22:23:11.062080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.061971    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:11.062198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062174    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.062414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062391    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.062490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062414    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062435    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062451    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062465    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn5jp\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-kube-api-access-dn5jp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062479    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062495    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062508    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062521    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494e40ea-8a23-4189-8191-0035bf7ed04f-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062535    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062563    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062577    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062584    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062591    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494e40ea-8a23-4189-8191-0035bf7ed04f-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062623    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062643    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlw4\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-kube-api-access-wwlw4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062658    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062673    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5355c-3025-47f3-8333-39439647415d-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.062919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.062686    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9fd184-8208-4667-b204-53d2e5e9581b-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.063635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.063200    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:11.063635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.063250    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.063635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.063393    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.064577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.064551    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:11.064745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.064717    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "kube-api-access-rf6q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:11.064838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.064819    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token" pod "0cbeca0a-5b04-4bbc-b01d-3f693635106a" (UID: "0cbeca0a-5b04-4bbc-b01d-3f693635106a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:11.065203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.065178    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "kube-api-access-g7j5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:11.065265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.065211    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:11.066881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.066863    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token" pod "52d5355c-3025-47f3-8333-39439647415d" (UID: "52d5355c-3025-47f3-8333-39439647415d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:11.163923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163888    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.163923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163919    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6q2\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-kube-api-access-rf6q2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163933    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163945    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5355c-3025-47f3-8333-39439647415d-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163958    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163968    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbeca0a-5b04-4bbc-b01d-3f693635106a-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163981    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.163995    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j5d\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-kube-api-access-g7j5d\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.164009    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbeca0a-5b04-4bbc-b01d-3f693635106a-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.164022    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5355c-3025-47f3-8333-39439647415d-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.164139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.164036    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5355c-3025-47f3-8333-39439647415d-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:11.235251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.235214    4410 generic.go:356] "Generic (PLEG): container finished" podID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerID="f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef" exitCode=0
Oct 30 22:23:11.235416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.235270    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"
Oct 30 22:23:11.235416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.235300    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" event={"ID":"494e40ea-8a23-4189-8191-0035bf7ed04f","Type":"ContainerDied","Data":"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"}
Oct 30 22:23:11.235416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.235363    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx" event={"ID":"494e40ea-8a23-4189-8191-0035bf7ed04f","Type":"ContainerDied","Data":"5a32646aa57d7cccfb14ac19f8bb5eae14631af2c8de4fe3376768c23fa0b34a"}
Oct 30 22:23:11.235416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.235390    4410 scope.go:122] "RemoveContainer" containerID="f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"
Oct 30 22:23:11.239057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.239025    4410 generic.go:356] "Generic (PLEG): container finished" podID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerID="696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11" exitCode=0
Oct 30 22:23:11.239205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.239059    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" event={"ID":"cc9fd184-8208-4667-b204-53d2e5e9581b","Type":"ContainerDied","Data":"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"}
Oct 30 22:23:11.239205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.239123    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk" event={"ID":"cc9fd184-8208-4667-b204-53d2e5e9581b","Type":"ContainerDied","Data":"559759e80d079116f7f72c98be43a42aaad977daa7dcee68e1c5a0682a7ed33c"}
Oct 30 22:23:11.239205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.239075    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"
Oct 30 22:23:11.240765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.240739    4410 generic.go:356] "Generic (PLEG): container finished" podID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerID="04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272" exitCode=0
Oct 30 22:23:11.240903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.240889    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" event={"ID":"0cbeca0a-5b04-4bbc-b01d-3f693635106a","Type":"ContainerDied","Data":"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"}
Oct 30 22:23:11.240964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.240928    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78" event={"ID":"0cbeca0a-5b04-4bbc-b01d-3f693635106a","Type":"ContainerDied","Data":"bc0f0c8a4903c36430afb8523551b7a0417a3a407f70a11d74d07557e40fc298"}
Oct 30 22:23:11.241024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.240998    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"
Oct 30 22:23:11.245878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.245855    4410 generic.go:356] "Generic (PLEG): container finished" podID="52d5355c-3025-47f3-8333-39439647415d" containerID="ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf" exitCode=0
Oct 30 22:23:11.245996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.245900    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" event={"ID":"52d5355c-3025-47f3-8333-39439647415d","Type":"ContainerDied","Data":"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"}
Oct 30 22:23:11.245996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.245933    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g" event={"ID":"52d5355c-3025-47f3-8333-39439647415d","Type":"ContainerDied","Data":"1c75c555b77eab8661a5ca9da52f965ad2071dcf226f3c87f0cee3dd13f7734e"}
Oct 30 22:23:11.246094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.245998    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"
Oct 30 22:23:11.251688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.251664    4410 scope.go:122] "RemoveContainer" containerID="f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"
Oct 30 22:23:11.252107 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:11.252079    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef\": container with ID starting with f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef not found: ID does not exist" containerID="f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"
Oct 30 22:23:11.252258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.252232    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef"} err="failed to get container status \"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef\": rpc error: code = NotFound desc = could not find container \"f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef\": container with ID starting with f5af6263e3792474a19b732681b1cbb1fe2d36d5c7f8f238147d2ac3db22fbef not found: ID does not exist"
Oct 30 22:23:11.252358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.252340    4410 scope.go:122] "RemoveContainer" containerID="696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"
Oct 30 22:23:11.263078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.263058    4410 scope.go:122] "RemoveContainer" containerID="696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"
Oct 30 22:23:11.263380 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:11.263354    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11\": container with ID starting with 696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11 not found: ID does not exist" containerID="696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"
Oct 30 22:23:11.263455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.263389    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11"} err="failed to get container status \"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11\": rpc error: code = NotFound desc = could not find container \"696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11\": container with ID starting with 696c5a36e09f51a186f759a78db30b53b06b3922eb52664b910821e929a50b11 not found: ID does not exist"
Oct 30 22:23:11.263455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.263405    4410 scope.go:122] "RemoveContainer" containerID="04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"
Oct 30 22:23:11.271927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.271909    4410 scope.go:122] "RemoveContainer" containerID="04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"
Oct 30 22:23:11.272196 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:11.272171    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272\": container with ID starting with 04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272 not found: ID does not exist" containerID="04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"
Oct 30 22:23:11.272242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.272206    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272"} err="failed to get container status \"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272\": rpc error: code = NotFound desc = could not find container \"04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272\": container with ID starting with 04215a984aa04ee44d5a96613f300dd212337940b06492950dcea44b64035272 not found: ID does not exist"
Oct 30 22:23:11.272242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.272229    4410 scope.go:122] "RemoveContainer" containerID="ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"
Oct 30 22:23:11.282712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.282691    4410 scope.go:122] "RemoveContainer" containerID="ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"
Oct 30 22:23:11.283023 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:11.282994    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf\": container with ID starting with ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf not found: ID does not exist" containerID="ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"
Oct 30 22:23:11.283086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.283033    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf"} err="failed to get container status \"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf\": rpc error: code = NotFound desc = could not find container \"ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf\": container with ID starting with ba181c0f3da28cb100fa93076a95326f4c7482392ed2fdad6c45c7c196dc4cbf not found: ID does not exist"
Oct 30 22:23:11.476239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.476177    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 22:23:11.478596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.478570    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-r6nlx"]
Oct 30 22:23:11.586628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.586600    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 22:23:11.592556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.592524    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fjr78"]
Oct 30 22:23:11.703864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.703829    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:23:11.704405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.704387    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j249g"]
Oct 30 22:23:11.811406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.811333    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:23:11.815981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:11.815955    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lzkmk"]
Oct 30 22:23:13.019149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.019113    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" path="/var/lib/kubelet/pods/0cbeca0a-5b04-4bbc-b01d-3f693635106a/volumes"
Oct 30 22:23:13.019590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.019576    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" path="/var/lib/kubelet/pods/494e40ea-8a23-4189-8191-0035bf7ed04f/volumes"
Oct 30 22:23:13.020012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.019998    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d5355c-3025-47f3-8333-39439647415d" path="/var/lib/kubelet/pods/52d5355c-3025-47f3-8333-39439647415d/volumes"
Oct 30 22:23:13.020938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.020918    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" path="/var/lib/kubelet/pods/cc9fd184-8208-4667-b204-53d2e5e9581b/volumes"
Oct 30 22:23:13.794752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.794715    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:13.983563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:13.983527    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:16.382877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:16.382832    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.028752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:17.028718    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.206705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:17.206668    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.479769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:18.479720    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.521613 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:20.521574    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:23:21.365499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:21.365456    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:21.622717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:21.622683    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:21.840037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:21.840002    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"]
Oct 30 22:23:21.888720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:21.888691    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d66ae6-thfvk"]
Oct 30 22:23:22.259460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:22.259341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:23.019649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:23.019610    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014b60ba-73ac-4802-8414-73984b145cc1" path="/var/lib/kubelet/pods/014b60ba-73ac-4802-8414-73984b145cc1/volumes"
Oct 30 22:23:23.741277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:23.741246    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:23.902229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:23.902193    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.081379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:24.081288    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.981137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:26.981102    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.159186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.159151    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.525298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525263    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:23:27.525717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525702    4410 cpu_manager.go:414] "containerMap: removing container" podUID="014b60ba-73ac-4802-8414-73984b145cc1" containerName="revoke-keys"
Oct 30 22:23:27.525762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525717    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b60ba-73ac-4802-8414-73984b145cc1" containerName="revoke-keys"
Oct 30 22:23:27.525762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525747    4410 cpu_manager.go:414] "containerMap: removing container" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy"
Oct 30 22:23:27.525762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525752    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy"
Oct 30 22:23:27.525762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525760    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy"
Oct 30 22:23:27.525916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525765    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy"
Oct 30 22:23:27.525916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525842    4410 cpu_manager.go:414] "containerMap: removing container" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy"
Oct 30 22:23:27.525916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525852    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy"
Oct 30 22:23:27.525916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525866    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy"
Oct 30 22:23:27.525916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525870    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy"
Oct 30 22:23:27.526071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525952    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d5355c-3025-47f3-8333-39439647415d" containerName="istio-proxy"
Oct 30 22:23:27.526071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525960    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbeca0a-5b04-4bbc-b01d-3f693635106a" containerName="istio-proxy"
Oct 30 22:23:27.526071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525966    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b60ba-73ac-4802-8414-73984b145cc1" containerName="revoke-keys"
Oct 30 22:23:27.526071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525974    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="494e40ea-8a23-4189-8191-0035bf7ed04f" containerName="istio-proxy"
Oct 30 22:23:27.526071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.525979    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9fd184-8208-4667-b204-53d2e5e9581b" containerName="istio-proxy"
Oct 30 22:23:27.529148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.529132    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.543237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.543214    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.543350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.543246    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xp55\" (UniqueName: \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.560985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.560948    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.633512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.633480    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-cce50a-dockercfg-2st55\""
Oct 30 22:23:27.633512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.633480    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-cce50a\""
Oct 30 22:23:27.636218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.636192    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:23:27.644558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.644530    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.644684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.644568    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xp55\" (UniqueName: \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.645341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.645311    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.701291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.701264    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xp55\" (UniqueName: \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\") pod \"payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:27.842473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:27.842416    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:28.214560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.214529    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:23:28.216812 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:23:28.216784    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5b463d_d07d_43c0_8018_25349551c737.slice/crio-54008db1e64e9f563d431e2453ce0b5050daefd543180a965ce8d1a2b22fc24d WatchSource:0}: Error finding container 54008db1e64e9f563d431e2453ce0b5050daefd543180a965ce8d1a2b22fc24d: Status 404 returned error can't find the container with id 54008db1e64e9f563d431e2453ce0b5050daefd543180a965ce8d1a2b22fc24d
Oct 30 22:23:28.358992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.358957    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" event={"ID":"1c5b463d-d07d-43c0-8018-25349551c737","Type":"ContainerStarted","Data":"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"}
Oct 30 22:23:28.359150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.359022    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" event={"ID":"1c5b463d-d07d-43c0-8018-25349551c737","Type":"ContainerStarted","Data":"54008db1e64e9f563d431e2453ce0b5050daefd543180a965ce8d1a2b22fc24d"}
Oct 30 22:23:28.359150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.359052    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:28.377405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.377381    4410 scope.go:122] "RemoveContainer" containerID="44a4377f370ed0efaa02102c2fb6409da7e0c72c0fbfb04cc70fea4dad3d2af1"
Oct 30 22:23:28.386012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.385989    4410 scope.go:122] "RemoveContainer" containerID="4256a1711d550896f1c8c466f5b976b18ac83abeaa88cade3646a2bb460c48e9"
Oct 30 22:23:28.475469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:28.475381    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" podStartSLOduration=1.4753673809999999 podStartE2EDuration="1.475367381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.475294172 +0000 UTC m=+2461.564031967" watchObservedRunningTime="2026-10-30 22:23:28.475367381 +0000 UTC m=+2461.564105184"
Oct 30 22:23:29.273605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:29.273573    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.437455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:29.437416    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.586315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:29.586239    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:23:29.586512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:29.586476    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" containerID="cri-o://d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a" gracePeriod=30
Oct 30 22:23:29.718595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:29.718554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.856510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:30.856470    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:32.437452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:32.437416    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:32.437847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:32.437465    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:34.858393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.858340    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:23:34.929657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929630    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.929751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929701    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.929877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929860    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.929954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929937    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929979    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.929938    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:34.930135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930091    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930125    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:34.930249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930147    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930183    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930202    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:34.930249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930238    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\") pod \"580c0cc4-6f99-47de-9070-d707b8a5fa85\" (UID: \"580c0cc4-6f99-47de-9070-d707b8a5fa85\") "
Oct 30 22:23:34.930459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930254    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:34.930595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930570    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:34.930678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930660    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.930738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930687    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/580c0cc4-6f99-47de-9070-d707b8a5fa85-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.930738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930703    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.930738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.930719    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.931993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.931967    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:34.932229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.932210    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:34.932624 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.932603    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:34.932712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:34.932621    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl" pod "580c0cc4-6f99-47de-9070-d707b8a5fa85" (UID: "580c0cc4-6f99-47de-9070-d707b8a5fa85"). InnerVolumeSpecName "kube-api-access-6d9sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:35.031892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.031865    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.032013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.031898    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9sl\" (UniqueName: \"kubernetes.io/projected/580c0cc4-6f99-47de-9070-d707b8a5fa85-kube-api-access-6d9sl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.032013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.031915    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.032013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.031928    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.032013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.031942    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/580c0cc4-6f99-47de-9070-d707b8a5fa85-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.079225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079194    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"]
Oct 30 22:23:35.079703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079685    4410 cpu_manager.go:414] "containerMap: removing container" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.079828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079705    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.079828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079750    4410 cpu_manager.go:414] "containerMap: removing container" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.079828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079758    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.080002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079904    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.080002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.079918    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerName="istio-proxy"
Oct 30 22:23:35.083205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.083183    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.133450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.133377    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.133607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.133580    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnzhc\" (UniqueName: \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.133731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.133705    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.135152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.135133    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:35.135251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.135222    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:35.135318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.135288    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:35.135393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.135372    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:23:35.141719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.141698    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"]
Oct 30 22:23:35.234950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.234921    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnzhc\" (UniqueName: \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.235099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.234970    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.235099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.235028    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.235622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.235600    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.237260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.237238    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.291911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.291890    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnzhc\" (UniqueName: \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\") pod \"maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.396110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.396083    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:35.402636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.402611    4410 generic.go:356] "Generic (PLEG): container finished" podID="580c0cc4-6f99-47de-9070-d707b8a5fa85" containerID="d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a" exitCode=0
Oct 30 22:23:35.402736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.402662    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"
Oct 30 22:23:35.402736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.402693    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerDied","Data":"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"}
Oct 30 22:23:35.402877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.402753    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm" event={"ID":"580c0cc4-6f99-47de-9070-d707b8a5fa85","Type":"ContainerDied","Data":"31fd7d3772b3002b9d8d5516604bcfee403c5c9148a8afe3c6bbaf3a5f9e8aa0"}
Oct 30 22:23:35.402877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.402804    4410 scope.go:122] "RemoveContainer" containerID="d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"
Oct 30 22:23:35.412447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.412424    4410 scope.go:122] "RemoveContainer" containerID="49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"
Oct 30 22:23:35.421230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.421199    4410 scope.go:122] "RemoveContainer" containerID="d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"
Oct 30 22:23:35.421551 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:35.421506    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a\": container with ID starting with d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a not found: ID does not exist" containerID="d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"
Oct 30 22:23:35.421627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.421550    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a"} err="failed to get container status \"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a\": rpc error: code = NotFound desc = could not find container \"d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a\": container with ID starting with d8c0814f45c15f3fadd1f153a8bdbf4057270d754b1fc60debeb46ba95d5898a not found: ID does not exist"
Oct 30 22:23:35.421627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.421568    4410 scope.go:122] "RemoveContainer" containerID="49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"
Oct 30 22:23:35.421888 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:35.421866    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57\": container with ID starting with 49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57 not found: ID does not exist" containerID="49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"
Oct 30 22:23:35.422842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.421896    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57"} err="failed to get container status \"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57\": rpc error: code = NotFound desc = could not find container \"49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57\": container with ID starting with 49bd294f8ba36a9b6927b2997f2dc2cd9c80b7efd6cab6b4d58e7ae5859dbd57 not found: ID does not exist"
Oct 30 22:23:35.568238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.568210    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:23:35.570587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.570566    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d66ae6-openshift-default-675d86cfd5-8xfpm"]
Oct 30 22:23:35.773530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.773505    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"]
Oct 30 22:23:35.775553 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:23:35.775522    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefad86c5_3bce_4c71_8729_18e4ad9f2b11.slice/crio-da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872 WatchSource:0}: Error finding container da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872: Status 404 returned error can't find the container with id da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872
Oct 30 22:23:35.812437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:35.812367    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:36.409578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:36.409541    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6" event={"ID":"efad86c5-3bce-4c71-8729-18e4ad9f2b11","Type":"ContainerStarted","Data":"31fad34cc0a9108442311948a76f27de5e35001c1f644945124be937ae68643b"}
Oct 30 22:23:36.410024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:36.409602    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6" event={"ID":"efad86c5-3bce-4c71-8729-18e4ad9f2b11","Type":"ContainerStarted","Data":"da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872"}
Oct 30 22:23:36.523804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:36.523740    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6" podStartSLOduration=2.523726108 podStartE2EDuration="2.523726108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:36.522112938 +0000 UTC m=+2469.610850741" watchObservedRunningTime="2026-10-30 22:23:36.523726108 +0000 UTC m=+2469.612463910"
Oct 30 22:23:37.019141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:37.019066    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580c0cc4-6f99-47de-9070-d707b8a5fa85" path="/var/lib/kubelet/pods/580c0cc4-6f99-47de-9070-d707b8a5fa85/volumes"
Oct 30 22:23:37.418488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:37.418455    4410 generic.go:356] "Generic (PLEG): container finished" podID="efad86c5-3bce-4c71-8729-18e4ad9f2b11" containerID="31fad34cc0a9108442311948a76f27de5e35001c1f644945124be937ae68643b" exitCode=0
Oct 30 22:23:37.418905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:37.418537    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6" event={"ID":"efad86c5-3bce-4c71-8729-18e4ad9f2b11","Type":"ContainerDied","Data":"31fad34cc0a9108442311948a76f27de5e35001c1f644945124be937ae68643b"}
Oct 30 22:23:38.568660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.568633    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:38.671465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.671431    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\") pod \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") "
Oct 30 22:23:38.671623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.671504    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\" (UniqueName: \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\") pod \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") "
Oct 30 22:23:38.671623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.671547    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\") pod \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\" (UID: \"efad86c5-3bce-4c71-8729-18e4ad9f2b11\") "
Oct 30 22:23:38.671762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.671738    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca" pod "efad86c5-3bce-4c71-8729-18e4ad9f2b11" (UID: "efad86c5-3bce-4c71-8729-18e4ad9f2b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:38.672014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.671990    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:38.673509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.673490    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca" pod "efad86c5-3bce-4c71-8729-18e4ad9f2b11" (UID: "efad86c5-3bce-4c71-8729-18e4ad9f2b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:38.673578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.673559    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc" pod "efad86c5-3bce-4c71-8729-18e4ad9f2b11" (UID: "efad86c5-3bce-4c71-8729-18e4ad9f2b11"). InnerVolumeSpecName "kube-api-access-rnzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:38.773488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.773455    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnzhc\" (UniqueName: \"kubernetes.io/projected/efad86c5-3bce-4c71-8729-18e4ad9f2b11-kube-api-access-rnzhc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:38.773488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:38.773483    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad86c5-3bce-4c71-8729-18e4ad9f2b11-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:39.371052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:39.370977    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:23:39.431578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:39.431552    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6" event={"ID":"efad86c5-3bce-4c71-8729-18e4ad9f2b11","Type":"ContainerDied","Data":"da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872"}
Oct 30 22:23:39.431578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:39.431572    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"
Oct 30 22:23:39.431759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:39.431597    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da54ecd5f6e6a489ca96093ef0302617011df36e9a273051b97616b23cd2c872"
Oct 30 22:23:41.045356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:41.045250    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:46.071379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:46.071334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:46.976458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:46.976417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:47.244901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:47.244763    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:50.522250 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:23:50.522211    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:23:51.038942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.038903    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:51.526511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.526481    4410 generic.go:356] "Generic (PLEG): container finished" podID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerID="051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5" exitCode=0
Oct 30 22:23:51.526905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.526522    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerDied","Data":"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"}
Oct 30 22:23:51.526905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.526904    4410 scope.go:122] "RemoveContainer" containerID="051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"
Oct 30 22:23:51.528240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.528211    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:51.528345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.528275    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:51.528345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:51.528306    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:52.533592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:52.533555    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerStarted","Data":"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"}
Oct 30 22:23:52.725253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:52.725218    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:23:52.726573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:52.726546    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:23:52.726708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:52.726606    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 22:23:53.726024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:53.725990    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:23:53.726406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:53.726069    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 22:23:54.726116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:54.726081    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:23:54.726560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:54.726144    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 22:23:55.725892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:55.725860    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:23:55.726081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:55.725912    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 22:23:56.301819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:56.301761    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:56.726293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:56.726254    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:23:56.726487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:56.726313    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 22:23:56.736415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:56.736382    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:23:57.729382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:57.729351    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:23:57.730440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:23:57.730418    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:24:00.462709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:00.462658    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:01.012348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:01.012291    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:01.226541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:01.226510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.124497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:02.124459    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.431185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:02.431153    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.606026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:02.605987    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.714188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:02.714110    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"]
Oct 30 22:24:02.763305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:02.763271    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2edb49-qgbdp"]
Oct 30 22:24:03.019676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:03.019595    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa678f3-e585-4b2e-9e2e-ef701951b959" path="/var/lib/kubelet/pods/6aa678f3-e585-4b2e-9e2e-ef701951b959/volumes"
Oct 30 22:24:03.144046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:03.144004    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:04.430364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:04.430330    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:24:04.430836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:04.430518    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" containerID="cri-o://0a9e5c4e79e95b423d5cefdca61cea8f611d581078204c1c9fd19ca7878b1340" gracePeriod=30
Oct 30 22:24:06.842874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:06.842839    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:24:06.896463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:06.896429    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:24:06.896610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:06.896489    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:24:07.002293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:07.002265    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:24:07.085292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:07.085256    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.408633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:07.408601    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.591230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:07.591196    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.788877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:07.788796    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:08.184439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:08.184384    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:09.656254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.656222    4410 generic.go:356] "Generic (PLEG): container finished" podID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerID="0a9e5c4e79e95b423d5cefdca61cea8f611d581078204c1c9fd19ca7878b1340" exitCode=0
Oct 30 22:24:09.656640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.656300    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" event={"ID":"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2","Type":"ContainerDied","Data":"0a9e5c4e79e95b423d5cefdca61cea8f611d581078204c1c9fd19ca7878b1340"}
Oct 30 22:24:09.687730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.687704    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:24:09.865495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865426    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.865495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865481    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.865751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865519    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.865751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865581    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.865751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865613    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.865751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865709    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865829    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865841    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865882    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865966    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865966    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:09.866012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.865969    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:09.866340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866029    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\") pod \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\" (UID: \"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2\") "
Oct 30 22:24:09.866735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866488    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.866735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866513    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.866735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866526    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.866735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866539    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.866735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.866698    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:09.868175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.868150    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:09.868282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.868170    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:09.869163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.869141    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:09.869460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.869437    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg" pod "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" (UID: "d9a85a03-e3a6-4c45-bce9-36b6bd4193a2"). InnerVolumeSpecName "kube-api-access-hn4kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:09.968285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.968251    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.968285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.968277    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.968285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.968287    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.968285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.968295    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:09.968651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:09.968305    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4kg\" (UniqueName: \"kubernetes.io/projected/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2-kube-api-access-hn4kg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.663508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:10.663471    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"
Oct 30 22:24:10.663508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:10.663497    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9" event={"ID":"d9a85a03-e3a6-4c45-bce9-36b6bd4193a2","Type":"ContainerDied","Data":"a51af0da2ba2425f13d2cbef40127b8b13911cf1f8d388d9b5a72d866827ea40"}
Oct 30 22:24:10.664039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:10.663561    4410 scope.go:122] "RemoveContainer" containerID="0a9e5c4e79e95b423d5cefdca61cea8f611d581078204c1c9fd19ca7878b1340"
Oct 30 22:24:10.879913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:10.878668    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:24:10.924450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:10.924383    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-5q9b9"]
Oct 30 22:24:11.019869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:11.019840    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" path="/var/lib/kubelet/pods/d9a85a03-e3a6-4c45-bce9-36b6bd4193a2/volumes"
Oct 30 22:24:11.843945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:11.843900    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" podUID="c3c15231-da63-4c54-af02-a15a52569ad0" containerName="payload-processing" containerID="cri-o://d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c" gracePeriod=25
Oct 30 22:24:12.002635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.002596    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" podUID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" containerName="payload-pre-processing" containerID="cri-o://92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b" gracePeriod=25
Oct 30 22:24:12.131115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.131094    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:24:12.257769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.257744    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:24:12.295709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.295679    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\") pod \"c3c15231-da63-4c54-af02-a15a52569ad0\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") "
Oct 30 22:24:12.295857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.295791    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\" (UniqueName: \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\") pod \"c3c15231-da63-4c54-af02-a15a52569ad0\" (UID: \"c3c15231-da63-4c54-af02-a15a52569ad0\") "
Oct 30 22:24:12.296099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.296078    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume" pod "c3c15231-da63-4c54-af02-a15a52569ad0" (UID: "c3c15231-da63-4c54-af02-a15a52569ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:12.296228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.296208    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c15231-da63-4c54-af02-a15a52569ad0-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:12.297734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.297714    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp" pod "c3c15231-da63-4c54-af02-a15a52569ad0" (UID: "c3c15231-da63-4c54-af02-a15a52569ad0"). InnerVolumeSpecName "kube-api-access-zf9vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:12.396870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.396834    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\") pod \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") "
Oct 30 22:24:12.397088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.396932    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\" (UniqueName: \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\") pod \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\" (UID: \"1ed13549-2bb8-430d-8f0c-c10916ea81b6\") "
Oct 30 22:24:12.397209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.397185    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume" pod "1ed13549-2bb8-430d-8f0c-c10916ea81b6" (UID: "1ed13549-2bb8-430d-8f0c-c10916ea81b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:12.397308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.397294    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed13549-2bb8-430d-8f0c-c10916ea81b6-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:12.397368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.397310    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf9vp\" (UniqueName: \"kubernetes.io/projected/c3c15231-da63-4c54-af02-a15a52569ad0-kube-api-access-zf9vp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:12.398955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.398929    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2" pod "1ed13549-2bb8-430d-8f0c-c10916ea81b6" (UID: "1ed13549-2bb8-430d-8f0c-c10916ea81b6"). InnerVolumeSpecName "kube-api-access-xwdf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:12.497916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.497892    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdf2\" (UniqueName: \"kubernetes.io/projected/1ed13549-2bb8-430d-8f0c-c10916ea81b6-kube-api-access-xwdf2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:12.581402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.581371    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.681223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.681137    4410 generic.go:356] "Generic (PLEG): container finished" podID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" containerID="92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b" exitCode=0
Oct 30 22:24:12.681223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.681182    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"
Oct 30 22:24:12.681425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.681219    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" event={"ID":"1ed13549-2bb8-430d-8f0c-c10916ea81b6","Type":"ContainerDied","Data":"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"}
Oct 30 22:24:12.681425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.681269    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm" event={"ID":"1ed13549-2bb8-430d-8f0c-c10916ea81b6","Type":"ContainerDied","Data":"938f384197660aebcec9b34dcec4a6da1309d561a5783a51b57585dc9ca8e69d"}
Oct 30 22:24:12.681425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.681293    4410 scope.go:122] "RemoveContainer" containerID="92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"
Oct 30 22:24:12.682886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.682855    4410 generic.go:356] "Generic (PLEG): container finished" podID="c3c15231-da63-4c54-af02-a15a52569ad0" containerID="d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c" exitCode=0
Oct 30 22:24:12.682989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.682894    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"
Oct 30 22:24:12.682989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.682909    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" event={"ID":"c3c15231-da63-4c54-af02-a15a52569ad0","Type":"ContainerDied","Data":"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"}
Oct 30 22:24:12.682989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.682946    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr" event={"ID":"c3c15231-da63-4c54-af02-a15a52569ad0","Type":"ContainerDied","Data":"b423d1a3b6919afc467319b1d69c4254564a974e48cd91d1d80eea5590801c00"}
Oct 30 22:24:12.692067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.692050    4410 scope.go:122] "RemoveContainer" containerID="92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"
Oct 30 22:24:12.692330 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:24:12.692311    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b\": container with ID starting with 92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b not found: ID does not exist" containerID="92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"
Oct 30 22:24:12.692400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.692338    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b"} err="failed to get container status \"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b\": rpc error: code = NotFound desc = could not find container \"92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b\": container with ID starting with 92008b9b4679ae66fad38bc74c74441a1f79e428ff41363c6e367ba8905ee17b not found: ID does not exist"
Oct 30 22:24:12.692400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.692357    4410 scope.go:122] "RemoveContainer" containerID="d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"
Oct 30 22:24:12.700680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.700661    4410 scope.go:122] "RemoveContainer" containerID="d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"
Oct 30 22:24:12.700937 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:24:12.700918    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c\": container with ID starting with d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c not found: ID does not exist" containerID="d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"
Oct 30 22:24:12.700997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.700944    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c"} err="failed to get container status \"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c\": rpc error: code = NotFound desc = could not find container \"d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c\": container with ID starting with d5846a3f3ecae6da2d0b593d3c65517bf8a0b5a2c368e8809a5a7da77c3f566c not found: ID does not exist"
Oct 30 22:24:12.846515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.846481    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:24:12.850407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.850384    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-56d4599cf5-ws6cm"]
Oct 30 22:24:12.960403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.960335    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:24:12.977786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:12.977751    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-85b4df78c9-wjvgr"]
Oct 30 22:24:13.019410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:13.019382    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" path="/var/lib/kubelet/pods/1ed13549-2bb8-430d-8f0c-c10916ea81b6/volumes"
Oct 30 22:24:13.019793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:13.019765    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c15231-da63-4c54-af02-a15a52569ad0" path="/var/lib/kubelet/pods/c3c15231-da63-4c54-af02-a15a52569ad0/volumes"
Oct 30 22:24:16.393587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:16.393553    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:16.757034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:16.756944    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:24:16.757234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:16.757180    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy" containerID="cri-o://b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299" gracePeriod=30
Oct 30 22:24:17.566981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:17.566925    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:18.415986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:18.415957    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:18.723614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:18.723531    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:19.659718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:19.659686    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.522193 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:24:20.522158    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:24:22.008236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.008210    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:24:22.098424 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098387    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098438    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098587    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098621    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098662    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098696    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098727    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.098859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098755    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098820    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.098859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098825    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:22.098995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098901    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\") pod \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\" (UID: \"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2\") "
Oct 30 22:24:22.099053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.098998    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.099113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099102    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.099242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099214    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.099445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099425    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.099445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099445    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.099588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099454    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.099588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099463    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.099588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.099472    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.100989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.100938    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:22.101090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.101040    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.101224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.101201    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.101266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.101213    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42" pod "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" (UID: "8b7e3514-caca-4abb-8a59-8b1aed8cc9c2"). InnerVolumeSpecName "kube-api-access-8gv42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.200760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.200733    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.200760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.200760    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gv42\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-kube-api-access-8gv42\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.200946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.200783    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.200946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.200793    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.751066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.751035    4410 generic.go:356] "Generic (PLEG): container finished" podID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerID="b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299" exitCode=0
Oct 30 22:24:22.751342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.751083    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"
Oct 30 22:24:22.751342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.751092    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" event={"ID":"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2","Type":"ContainerDied","Data":"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"}
Oct 30 22:24:22.751342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.751134    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc" event={"ID":"8b7e3514-caca-4abb-8a59-8b1aed8cc9c2","Type":"ContainerDied","Data":"ae0bcc130123846856927922831ef79d6e15b5607d966cec3d2d38feebd1fd1a"}
Oct 30 22:24:22.751342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.751153    4410 scope.go:122] "RemoveContainer" containerID="b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"
Oct 30 22:24:22.760968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.760944    4410 scope.go:122] "RemoveContainer" containerID="b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"
Oct 30 22:24:22.761213 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:24:22.761190    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299\": container with ID starting with b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299 not found: ID does not exist" containerID="b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"
Oct 30 22:24:22.761286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.761227    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299"} err="failed to get container status \"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299\": rpc error: code = NotFound desc = could not find container \"b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299\": container with ID starting with b548105b42cd85851d4107c3dfbbd62c10ebe42c52f4c4d6b7098239ee55b299 not found: ID does not exist"
Oct 30 22:24:22.920481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.920452    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:22.968275 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:22.968251    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:24:23.020831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:23.020754    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-zn9bc"]
Oct 30 22:24:23.104879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:23.104843    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.318949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:23.318845    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:24.812499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:24.812454    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:25.020683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:25.020650    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" path="/var/lib/kubelet/pods/8b7e3514-caca-4abb-8a59-8b1aed8cc9c2/volumes"
Oct 30 22:24:25.046420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:25.046391    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:27.927677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:27.927635    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.087635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:28.087596    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.246423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:28.246343    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.172539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:30.172512    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.777023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:36.776982    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:40.980806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:40.980753    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.541525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:42.541487    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.760475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:42.760437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.933089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:42.933052    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.020549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:43.020514    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"]
Oct 30 22:24:43.063377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:43.063343    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fe2565-8dmw6"]
Oct 30 22:24:43.422727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:43.422696    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.019039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:45.019006    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efad86c5-3bce-4c71-8729-18e4ad9f2b11" path="/var/lib/kubelet/pods/efad86c5-3bce-4c71-8729-18e4ad9f2b11/volumes"
Oct 30 22:24:45.372950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:45.372864    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.899129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:45.899098    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:48.115241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:48.115206    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:49.640452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:49.640420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:49.806307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:49.806274    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:49.969928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:49.969844    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.520858 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:24:50.520821    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:24:51.720309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:51.720269    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.019569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:52.019486    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.186510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:52.186470    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.362722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:52.362642    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.197561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.197524    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"]
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.197961    4410 cpu_manager.go:414] "containerMap: removing container" podUID="efad86c5-3bce-4c71-8729-18e4ad9f2b11" containerName="revoke-keys"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.197973    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efad86c5-3bce-4c71-8729-18e4ad9f2b11" containerName="revoke-keys"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.197983    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.197988    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198000    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" containerName="payload-pre-processing"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198005    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" containerName="payload-pre-processing"
Oct 30 22:24:53.198016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198018    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198023    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198038    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c3c15231-da63-4c54-af02-a15a52569ad0" containerName="payload-processing"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198043    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c15231-da63-4c54-af02-a15a52569ad0" containerName="payload-processing"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198112    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="efad86c5-3bce-4c71-8729-18e4ad9f2b11" containerName="revoke-keys"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198119    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c15231-da63-4c54-af02-a15a52569ad0" containerName="payload-processing"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198127    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7e3514-caca-4abb-8a59-8b1aed8cc9c2" containerName="istio-proxy"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198135    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed13549-2bb8-430d-8f0c-c10916ea81b6" containerName="payload-pre-processing"
Oct 30 22:24:53.198250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.198142    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a85a03-e3a6-4c45-bce9-36b6bd4193a2" containerName="istio-proxy"
Oct 30 22:24:53.202627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.202608    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.228754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.228723    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qfn\" (UniqueName: \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.228886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.228845    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.228941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.228928    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.306137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.306104    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:53.306137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.306124    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:53.306395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.306171    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:53.306395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.306279    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:24:53.308769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.308741    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"]
Oct 30 22:24:53.329728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.329708    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.329835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.329758    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qfn\" (UniqueName: \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.329896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.329844    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.330401 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.330373    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.332193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.332174    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.388598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.388570    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qfn\" (UniqueName: \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.516148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.516076    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:53.896058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.896029    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"]
Oct 30 22:24:53.897822 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:24:53.897792    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0b4dc8_1a9e_49a6_ac5e_caada5e8fa33.slice/crio-b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34 WatchSource:0}: Error finding container b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34: Status 404 returned error can't find the container with id b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34
Oct 30 22:24:53.930331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:53.930302    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87" event={"ID":"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33","Type":"ContainerStarted","Data":"b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34"}
Oct 30 22:24:54.937387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:54.937357    4410 generic.go:356] "Generic (PLEG): container finished" podID="fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" containerID="176c18d90fbb8aaf4a64485c5eb8c7f82cef83c974c4219698984c74a5716442" exitCode=0
Oct 30 22:24:54.937794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:54.937450    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87" event={"ID":"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33","Type":"ContainerDied","Data":"176c18d90fbb8aaf4a64485c5eb8c7f82cef83c974c4219698984c74a5716442"}
Oct 30 22:24:55.094484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:55.094452    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:56.084977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.084948    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:56.158787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.158749    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\") pod \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") "
Oct 30 22:24:56.158944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.158895    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\" (UniqueName: \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\") pod \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") "
Oct 30 22:24:56.159014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.158959    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\") pod \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\" (UID: \"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33\") "
Oct 30 22:24:56.159359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.159331    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca" pod "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" (UID: "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:56.161029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.161005    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn" pod "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" (UID: "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33"). InnerVolumeSpecName "kube-api-access-s5qfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:56.161146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.161077    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca" pod "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" (UID: "fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:56.260800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.260712    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qfn\" (UniqueName: \"kubernetes.io/projected/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-kube-api-access-s5qfn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.260800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.260739    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.260800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.260750    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.950254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.950225    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"
Oct 30 22:24:56.950254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.950232    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87" event={"ID":"fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33","Type":"ContainerDied","Data":"b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34"}
Oct 30 22:24:56.950517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:56.950275    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6ce94dca6976e7c1a2c659820a46741afeb4a3b127fcc714ca506861dba0e34"
Oct 30 22:24:58.275516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.275474    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"]
Oct 30 22:24:58.276204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.276183    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" containerName="revoke-keys"
Oct 30 22:24:58.276254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.276207    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" containerName="revoke-keys"
Oct 30 22:24:58.276321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.276310    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" containerName="revoke-keys"
Oct 30 22:24:58.279946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.279926    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.336365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.336317    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"]
Oct 30 22:24:58.381330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.381299    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2wk\" (UniqueName: \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.381488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.381415    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.381488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.381466    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.382988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.382966    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:58.383110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.383035    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:58.383110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.383037    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:58.386741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.386725    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:24:58.482762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.482735    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.482877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.482804    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.482926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.482909    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2wk\" (UniqueName: \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.483460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.483439    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.485005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.484985    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.539717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.539667    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2wk\" (UniqueName: \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\") pod \"maas-api-revoke-sub-0265f16f-mkqpb\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.592974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.592938    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:24:58.970623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:58.970592    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"]
Oct 30 22:24:58.972523 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:24:58.972496    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0601e5_59cf_4418_a24c_c25f4f3f015b.slice/crio-c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350 WatchSource:0}: Error finding container c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350: Status 404 returned error can't find the container with id c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350
Oct 30 22:24:59.970035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:59.970000    4410 generic.go:356] "Generic (PLEG): container finished" podID="4e0601e5-59cf-4418-a24c-c25f4f3f015b" containerID="2a5c3ce92d3f20306dc3630453460775258b6ac932bfa2d0c0d5b9f6a2d1fa84" exitCode=0
Oct 30 22:24:59.970428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:59.970088    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb" event={"ID":"4e0601e5-59cf-4418-a24c-c25f4f3f015b","Type":"ContainerDied","Data":"2a5c3ce92d3f20306dc3630453460775258b6ac932bfa2d0c0d5b9f6a2d1fa84"}
Oct 30 22:24:59.970428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:24:59.970137    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb" event={"ID":"4e0601e5-59cf-4418-a24c-c25f4f3f015b","Type":"ContainerStarted","Data":"c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350"}
Oct 30 22:25:00.351569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:00.351538    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:01.118002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.117977    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:25:01.211613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.211586    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\" (UniqueName: \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\") pod \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") "
Oct 30 22:25:01.211760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.211634    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\") pod \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") "
Oct 30 22:25:01.211833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.211808    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\") pod \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\" (UID: \"4e0601e5-59cf-4418-a24c-c25f4f3f015b\") "
Oct 30 22:25:01.212095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.212073    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca" pod "4e0601e5-59cf-4418-a24c-c25f4f3f015b" (UID: "4e0601e5-59cf-4418-a24c-c25f4f3f015b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:01.212253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.212238    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:01.213615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.213595    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca" pod "4e0601e5-59cf-4418-a24c-c25f4f3f015b" (UID: "4e0601e5-59cf-4418-a24c-c25f4f3f015b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:01.213697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.213647    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk" pod "4e0601e5-59cf-4418-a24c-c25f4f3f015b" (UID: "4e0601e5-59cf-4418-a24c-c25f4f3f015b"). InnerVolumeSpecName "kube-api-access-zp2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:01.313131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.313062    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp2wk\" (UniqueName: \"kubernetes.io/projected/4e0601e5-59cf-4418-a24c-c25f4f3f015b-kube-api-access-zp2wk\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:01.313131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.313089    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0601e5-59cf-4418-a24c-c25f4f3f015b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:01.983956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.983924    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb" event={"ID":"4e0601e5-59cf-4418-a24c-c25f4f3f015b","Type":"ContainerDied","Data":"c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350"}
Oct 30 22:25:01.984119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.983965    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"
Oct 30 22:25:01.984119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:01.983972    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29650764342bda6cc48e2dcbf505d3442df6ccfc6e860ce4336396a6828d350"
Oct 30 22:25:05.142746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:05.142710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.169424 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.169331    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.365158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.365119    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"]
Oct 30 22:25:10.365607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.365587    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4e0601e5-59cf-4418-a24c-c25f4f3f015b" containerName="revoke-keys"
Oct 30 22:25:10.365711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.365609    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0601e5-59cf-4418-a24c-c25f4f3f015b" containerName="revoke-keys"
Oct 30 22:25:10.365812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.365787    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0601e5-59cf-4418-a24c-c25f4f3f015b" containerName="revoke-keys"
Oct 30 22:25:10.369013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.368989    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.426239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.426171    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"]
Oct 30 22:25:10.446837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.446815    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:10.446958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.446856    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:25:10.446958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.446861    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:10.472447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.472425    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:10.504486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.504460    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbvb\" (UniqueName: \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.504574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.504508    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.504641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.504622    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.605727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.605703    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.605886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.605770    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.605952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.605903    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbvb\" (UniqueName: \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.606474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.606453    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.608049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.608029    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.662435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.662400    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbvb\" (UniqueName: \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:10.682102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:10.682052    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:11.055352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:11.055323    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"]
Oct 30 22:25:11.057450 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:25:11.057424    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01fb310_bace_43bb_8e5b_12ae43e4e87d.slice/crio-e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787 WatchSource:0}: Error finding container e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787: Status 404 returned error can't find the container with id e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787
Oct 30 22:25:12.040505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:12.040466    4410 generic.go:356] "Generic (PLEG): container finished" podID="f01fb310-bace-43bb-8e5b-12ae43e4e87d" containerID="423bcc3a2bcef9c8fe2b65d0fdb3e1540d7c0150374be7b3f1c53cf43c5ebeba" exitCode=0
Oct 30 22:25:12.040977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:12.040554    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s" event={"ID":"f01fb310-bace-43bb-8e5b-12ae43e4e87d","Type":"ContainerDied","Data":"423bcc3a2bcef9c8fe2b65d0fdb3e1540d7c0150374be7b3f1c53cf43c5ebeba"}
Oct 30 22:25:12.040977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:12.040658    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s" event={"ID":"f01fb310-bace-43bb-8e5b-12ae43e4e87d","Type":"ContainerStarted","Data":"e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787"}
Oct 30 22:25:13.176182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.176155    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:13.334043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.333954    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\" (UniqueName: \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\") pod \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") "
Oct 30 22:25:13.334043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.334022    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\") pod \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") "
Oct 30 22:25:13.334281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.334161    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\") pod \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\" (UID: \"f01fb310-bace-43bb-8e5b-12ae43e4e87d\") "
Oct 30 22:25:13.334511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.334488    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca" pod "f01fb310-bace-43bb-8e5b-12ae43e4e87d" (UID: "f01fb310-bace-43bb-8e5b-12ae43e4e87d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:13.336023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.336001    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca" pod "f01fb310-bace-43bb-8e5b-12ae43e4e87d" (UID: "f01fb310-bace-43bb-8e5b-12ae43e4e87d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:13.336141 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.336062    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb" pod "f01fb310-bace-43bb-8e5b-12ae43e4e87d" (UID: "f01fb310-bace-43bb-8e5b-12ae43e4e87d"). InnerVolumeSpecName "kube-api-access-nhbvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:13.435808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.435756    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbvb\" (UniqueName: \"kubernetes.io/projected/f01fb310-bace-43bb-8e5b-12ae43e4e87d-kube-api-access-nhbvb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.435808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.435806    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.435983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:13.435818    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01fb310-bace-43bb-8e5b-12ae43e4e87d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.051936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:14.051909    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"
Oct 30 22:25:14.052116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:14.051949    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s" event={"ID":"f01fb310-bace-43bb-8e5b-12ae43e4e87d","Type":"ContainerDied","Data":"e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787"}
Oct 30 22:25:14.052116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:14.051998    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34e4273758edafd3b261f5d923df60eb288b98220dfdd6b2c10e111a5ee5787"
Oct 30 22:25:14.803519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:14.803483    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:15.449578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:15.449543    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:15.618029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:15.617994    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.538841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.538796    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.901321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.901290    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.956279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.956250    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"]
Oct 30 22:25:16.956728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.956712    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f01fb310-bace-43bb-8e5b-12ae43e4e87d" containerName="revoke-keys"
Oct 30 22:25:16.956803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.956731    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01fb310-bace-43bb-8e5b-12ae43e4e87d" containerName="revoke-keys"
Oct 30 22:25:16.956869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.956857    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01fb310-bace-43bb-8e5b-12ae43e4e87d" containerName="revoke-keys"
Oct 30 22:25:16.962092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:16.962069    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.015235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.015211    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:17.064434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.064411    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:25:17.064573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.064437    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:17.064573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.064450    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:17.067011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.066985    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"]
Oct 30 22:25:17.067401 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.067379    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.067527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.067515    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.067605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.067588    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhrs\" (UniqueName: \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.086686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.086661    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"]
Oct 30 22:25:17.091601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.091579    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96567a-fbg87"]
Oct 30 22:25:17.104822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.104802    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:17.168685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.168627    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.168826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.168801    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.168900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.168835    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhrs\" (UniqueName: \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.169513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.169490    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.171191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.171173    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.236582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.236555    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhrs\" (UniqueName: \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\") pod \"maas-api-revoke-sub-ca986360-z6sss\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.274394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.274367    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:17.560566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.560492    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:17.680306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:17.680274    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"]
Oct 30 22:25:17.694997 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:25:17.694962    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62715d1d_70a7_439e_b281_50ba9f9c646d.slice/crio-508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f WatchSource:0}: Error finding container 508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f: Status 404 returned error can't find the container with id 508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f
Oct 30 22:25:18.082797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:18.082738    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss" event={"ID":"62715d1d-70a7-439e-b281-50ba9f9c646d","Type":"ContainerStarted","Data":"98898a27fd35922db1d477eceaa8d86d120c9f0e14b0d2310604f65b0530338d"}
Oct 30 22:25:18.082959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:18.082820    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss" event={"ID":"62715d1d-70a7-439e-b281-50ba9f9c646d","Type":"ContainerStarted","Data":"508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f"}
Oct 30 22:25:18.201131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:18.201084    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss" podStartSLOduration=2.201071378 podStartE2EDuration="2.201071378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:18.198468216 +0000 UTC m=+2571.287206044" watchObservedRunningTime="2026-10-30 22:25:18.201071378 +0000 UTC m=+2571.289809180"
Oct 30 22:25:19.020416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:19.020382    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33" path="/var/lib/kubelet/pods/fa0b4dc8-1a9e-49a6-ac5e-caada5e8fa33/volumes"
Oct 30 22:25:19.089831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:19.089766    4410 generic.go:356] "Generic (PLEG): container finished" podID="62715d1d-70a7-439e-b281-50ba9f9c646d" containerID="98898a27fd35922db1d477eceaa8d86d120c9f0e14b0d2310604f65b0530338d" exitCode=0
Oct 30 22:25:19.089997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:19.089827    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss" event={"ID":"62715d1d-70a7-439e-b281-50ba9f9c646d","Type":"ContainerDied","Data":"98898a27fd35922db1d477eceaa8d86d120c9f0e14b0d2310604f65b0530338d"}
Oct 30 22:25:20.234395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.234367    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:20.403017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.402987    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\" (UniqueName: \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\") pod \"62715d1d-70a7-439e-b281-50ba9f9c646d\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") "
Oct 30 22:25:20.403177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.403038    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\") pod \"62715d1d-70a7-439e-b281-50ba9f9c646d\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") "
Oct 30 22:25:20.403177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.403124    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\") pod \"62715d1d-70a7-439e-b281-50ba9f9c646d\" (UID: \"62715d1d-70a7-439e-b281-50ba9f9c646d\") "
Oct 30 22:25:20.403481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.403456    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca" pod "62715d1d-70a7-439e-b281-50ba9f9c646d" (UID: "62715d1d-70a7-439e-b281-50ba9f9c646d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:20.405119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.405099    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca" pod "62715d1d-70a7-439e-b281-50ba9f9c646d" (UID: "62715d1d-70a7-439e-b281-50ba9f9c646d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:20.405234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.405218    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs" pod "62715d1d-70a7-439e-b281-50ba9f9c646d" (UID: "62715d1d-70a7-439e-b281-50ba9f9c646d"). InnerVolumeSpecName "kube-api-access-tfhrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:20.504511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.504488    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:20.504511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.504510    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhrs\" (UniqueName: \"kubernetes.io/projected/62715d1d-70a7-439e-b281-50ba9f9c646d-kube-api-access-tfhrs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:20.504654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:20.504521    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62715d1d-70a7-439e-b281-50ba9f9c646d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:20.521091 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:25:20.521054    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:25:21.102298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:21.102267    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"
Oct 30 22:25:21.102470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:21.102309    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss" event={"ID":"62715d1d-70a7-439e-b281-50ba9f9c646d","Type":"ContainerDied","Data":"508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f"}
Oct 30 22:25:21.102470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:21.102358    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="508d115c91a7bba35fa9d4d2ab68a45d8e952163bc895422cfad2769fe363b1f"
Oct 30 22:25:21.894019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:21.893988    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.108743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:22.108706    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.964476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:23.964440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:24.158455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:24.158422    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:24.408875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:24.408827    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:25.814426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:25.814392    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.784895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:26.784858    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.962653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:26.962608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:27.139883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:27.139796    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.307566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:28.307518    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.461037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:28.461010    4410 scope.go:122] "RemoveContainer" containerID="50337cf669d922be755b82aaa81d1c16325e1c1f092a9faade80ebb658c84cc9"
Oct 30 22:25:31.282674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:31.282642    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:31.828587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:31.828551    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.031288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:32.031252    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.983509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:32.983474    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.256536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:33.256454    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.414397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:33.414359    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.456026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:33.455993    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"]
Oct 30 22:25:33.505258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:33.505227    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8b1e0a-7kj6s"]
Oct 30 22:25:33.862470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:33.862436    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.018525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:35.018495    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01fb310-bace-43bb-8e5b-12ae43e4e87d" path="/var/lib/kubelet/pods/f01fb310-bace-43bb-8e5b-12ae43e4e87d/volumes"
Oct 30 22:25:36.197360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:36.197327    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.373941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:37.373909    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.981312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:37.981273    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:38.291522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:38.291439    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:38.642248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:38.642219    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:40.145223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:40.145190    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:40.344531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:40.344490    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:40.445300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:40.445209    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:25:40.445495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:40.445470    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" containerID="cri-o://f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0" gracePeriod=30
Oct 30 22:25:40.499133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:40.499103    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:41.249104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.249073    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"]
Oct 30 22:25:41.249552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.249535    4410 cpu_manager.go:414] "containerMap: removing container" podUID="62715d1d-70a7-439e-b281-50ba9f9c646d" containerName="revoke-keys"
Oct 30 22:25:41.249723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.249553    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62715d1d-70a7-439e-b281-50ba9f9c646d" containerName="revoke-keys"
Oct 30 22:25:41.249723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.249629    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="62715d1d-70a7-439e-b281-50ba9f9c646d" containerName="revoke-keys"
Oct 30 22:25:41.254044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.254027    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.310364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.310335    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"]
Oct 30 22:25:41.329711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.329690    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.329868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.329749    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.329947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.329920    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgq8b\" (UniqueName: \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.357136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.357110    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:41.357136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.357116    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:41.357295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.357124    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:41.357295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.357190    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:25:41.430900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.430870    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.431039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.430924    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgq8b\" (UniqueName: \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.431039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.431001    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.431526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.431492    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.433231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.433210    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.487478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.487449    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgq8b\" (UniqueName: \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\") pod \"maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.568818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.568721    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:41.940100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:41.940074    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"]
Oct 30 22:25:41.941996 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:25:41.941966    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aab9c77_301f_4949_a117_c7f2f3cdbbb7.slice/crio-830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af WatchSource:0}: Error finding container 830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af: Status 404 returned error can't find the container with id 830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af
Oct 30 22:25:42.227878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:42.227811    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs" event={"ID":"7aab9c77-301f-4949-a117-c7f2f3cdbbb7","Type":"ContainerStarted","Data":"830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af"}
Oct 30 22:25:42.232872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:42.232844    4410 patch_prober.go:29] interesting pod/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:42.232991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:42.232898    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:43.235251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.235216    4410 generic.go:356] "Generic (PLEG): container finished" podID="7aab9c77-301f-4949-a117-c7f2f3cdbbb7" containerID="0037bea7a5e05e8e74c555d860ba2cb3b2fc80b9997e9e0f59f324192775e9c0" exitCode=0
Oct 30 22:25:43.235646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.235284    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs" event={"ID":"7aab9c77-301f-4949-a117-c7f2f3cdbbb7","Type":"ContainerDied","Data":"0037bea7a5e05e8e74c555d860ba2cb3b2fc80b9997e9e0f59f324192775e9c0"}
Oct 30 22:25:43.265313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.265285    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"]
Oct 30 22:25:43.268768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.268751    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.326386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.326358    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"]
Oct 30 22:25:43.350086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.350061    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.350216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.350104    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbq7\" (UniqueName: \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.350216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.350192    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.451456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.451422    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.451602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.451475    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbq7\" (UniqueName: \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.451602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.451528    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.452144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.452121    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.453938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.453908    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.511210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.511149    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbq7\" (UniqueName: \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\") pod \"maas-api-revoke-sub-df6ffadc-g6qzm\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.581074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.581049    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:43.929734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.929698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.954621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:43.954585    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"]
Oct 30 22:25:43.957017 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:25:43.956991    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05497d8_b999_4ba1_9449_79676d5d3da1.slice/crio-0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507 WatchSource:0}: Error finding container 0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507: Status 404 returned error can't find the container with id 0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507
Oct 30 22:25:44.242412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.242325    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm" event={"ID":"d05497d8-b999-4ba1-9449-79676d5d3da1","Type":"ContainerStarted","Data":"0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507"}
Oct 30 22:25:44.370406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.370382    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:44.464204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.464180    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\") pod \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") "
Oct 30 22:25:44.464337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.464251    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\" (UniqueName: \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\") pod \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") "
Oct 30 22:25:44.464337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.464314    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\") pod \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\" (UID: \"7aab9c77-301f-4949-a117-c7f2f3cdbbb7\") "
Oct 30 22:25:44.464657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.464628    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca" pod "7aab9c77-301f-4949-a117-c7f2f3cdbbb7" (UID: "7aab9c77-301f-4949-a117-c7f2f3cdbbb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:44.464842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.464822    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:44.466282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.466260    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca" pod "7aab9c77-301f-4949-a117-c7f2f3cdbbb7" (UID: "7aab9c77-301f-4949-a117-c7f2f3cdbbb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:44.466358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.466311    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b" pod "7aab9c77-301f-4949-a117-c7f2f3cdbbb7" (UID: "7aab9c77-301f-4949-a117-c7f2f3cdbbb7"). InnerVolumeSpecName "kube-api-access-rgq8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:44.565592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.565516    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:44.565592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:44.565554    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgq8b\" (UniqueName: \"kubernetes.io/projected/7aab9c77-301f-4949-a117-c7f2f3cdbbb7-kube-api-access-rgq8b\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.249098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.249065    4410 generic.go:356] "Generic (PLEG): container finished" podID="d05497d8-b999-4ba1-9449-79676d5d3da1" containerID="a938843f1e3b5a18557be688945d9d8aa67061b4063d25e2fae7dce5b5b0e513" exitCode=0
Oct 30 22:25:45.249493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.249156    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm" event={"ID":"d05497d8-b999-4ba1-9449-79676d5d3da1","Type":"ContainerDied","Data":"a938843f1e3b5a18557be688945d9d8aa67061b4063d25e2fae7dce5b5b0e513"}
Oct 30 22:25:45.250477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.250441    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs" event={"ID":"7aab9c77-301f-4949-a117-c7f2f3cdbbb7","Type":"ContainerDied","Data":"830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af"}
Oct 30 22:25:45.250575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.250489    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830c6dee13bd4ab11576241d2e25ee675bdf9b23c457e641d3a22829e39bb5af"
Oct 30 22:25:45.250575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.250462    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"
Oct 30 22:25:45.695512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.695489    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:25:45.778240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778213    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778289    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778360    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778410    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778443    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778471    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778489    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:45.778657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778536    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778562    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:45.778657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778579    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.778657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778607    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\") pod \"ba18dc36-4920-49b2-a111-498e08aa3344\" (UID: \"ba18dc36-4920-49b2-a111-498e08aa3344\") "
Oct 30 22:25:45.779037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.779009    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:45.779037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.778959    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:45.779171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.779124    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.779171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.779144    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.779285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.779166    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:45.780571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.780540    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:45.780683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.780665    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "kube-api-access-bg69l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:45.780916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.780893    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:45.780989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.780896    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token" pod "ba18dc36-4920-49b2-a111-498e08aa3344" (UID: "ba18dc36-4920-49b2-a111-498e08aa3344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:45.837370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.837285    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.880081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880052    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880078    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg69l\" (UniqueName: \"kubernetes.io/projected/ba18dc36-4920-49b2-a111-498e08aa3344-kube-api-access-bg69l\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880089    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba18dc36-4920-49b2-a111-498e08aa3344-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880100    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba18dc36-4920-49b2-a111-498e08aa3344-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880109    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880118    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:45.880227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:45.880126    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba18dc36-4920-49b2-a111-498e08aa3344-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:46.056450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.056413    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.257558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.257527    4410 generic.go:356] "Generic (PLEG): container finished" podID="ba18dc36-4920-49b2-a111-498e08aa3344" containerID="f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0" exitCode=0
Oct 30 22:25:46.258070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.257585    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"
Oct 30 22:25:46.258070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.257610    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerDied","Data":"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"}
Oct 30 22:25:46.258070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.257664    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl" event={"ID":"ba18dc36-4920-49b2-a111-498e08aa3344","Type":"ContainerDied","Data":"d214b535b78eef1db4a98dadd2fabab0c7718154b03a4d23ad393a68063d2e46"}
Oct 30 22:25:46.258070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.257688    4410 scope.go:122] "RemoveContainer" containerID="f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"
Oct 30 22:25:46.267965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.267946    4410 scope.go:122] "RemoveContainer" containerID="06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"
Oct 30 22:25:46.277552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.277532    4410 scope.go:122] "RemoveContainer" containerID="f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"
Oct 30 22:25:46.277852 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:25:46.277830    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0\": container with ID starting with f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0 not found: ID does not exist" containerID="f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"
Oct 30 22:25:46.277935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.277863    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0"} err="failed to get container status \"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0\": rpc error: code = NotFound desc = could not find container \"f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0\": container with ID starting with f62300bc29bdd877057e0547d7fe32cafc3d9d8df420d9c106339575be33cdd0 not found: ID does not exist"
Oct 30 22:25:46.277935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.277882    4410 scope.go:122] "RemoveContainer" containerID="06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"
Oct 30 22:25:46.278163 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:25:46.278144    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335\": container with ID starting with 06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335 not found: ID does not exist" containerID="06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"
Oct 30 22:25:46.278229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.278177    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335"} err="failed to get container status \"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335\": rpc error: code = NotFound desc = could not find container \"06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335\": container with ID starting with 06178b93a6461186e47c017e34207e67e6dd13576e159a01244da5264492d335 not found: ID does not exist"
Oct 30 22:25:46.397962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.397936    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:46.421131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.421103    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:25:46.471949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.471920    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b1e0a-openshift-default-5f759d897-xj4kl"]
Oct 30 22:25:46.486276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.486254    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\" (UniqueName: \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\") pod \"d05497d8-b999-4ba1-9449-79676d5d3da1\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") "
Oct 30 22:25:46.486446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.486433    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\") pod \"d05497d8-b999-4ba1-9449-79676d5d3da1\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") "
Oct 30 22:25:46.486499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.486477    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\") pod \"d05497d8-b999-4ba1-9449-79676d5d3da1\" (UID: \"d05497d8-b999-4ba1-9449-79676d5d3da1\") "
Oct 30 22:25:46.486918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.486892    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca" pod "d05497d8-b999-4ba1-9449-79676d5d3da1" (UID: "d05497d8-b999-4ba1-9449-79676d5d3da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:46.488262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.488244    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7" pod "d05497d8-b999-4ba1-9449-79676d5d3da1" (UID: "d05497d8-b999-4ba1-9449-79676d5d3da1"). InnerVolumeSpecName "kube-api-access-vfbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:46.488375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.488362    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca" pod "d05497d8-b999-4ba1-9449-79676d5d3da1" (UID: "d05497d8-b999-4ba1-9449-79676d5d3da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:46.588246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.588171    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:46.588246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.588199    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbq7\" (UniqueName: \"kubernetes.io/projected/d05497d8-b999-4ba1-9449-79676d5d3da1-kube-api-access-vfbq7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:46.588246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:46.588212    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05497d8-b999-4ba1-9449-79676d5d3da1-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:47.019067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:47.019032    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" path="/var/lib/kubelet/pods/ba18dc36-4920-49b2-a111-498e08aa3344/volumes"
Oct 30 22:25:47.265122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:47.265089    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"
Oct 30 22:25:47.265631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:47.265135    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm" event={"ID":"d05497d8-b999-4ba1-9449-79676d5d3da1","Type":"ContainerDied","Data":"0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507"}
Oct 30 22:25:47.265631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:47.265190    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa70c37527be755e3856af967c5f280b84f17c8bc806a0b5964b40a24948507"
Oct 30 22:25:50.521431 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:25:50.521396    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:25:50.829702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:50.829627    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.456935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:51.456893    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.675618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:51.675584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.466351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:52.466313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.681372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:52.681341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.749753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:54.749716    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.967909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:54.967878    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.418485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:56.418448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.684694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:25:57.684650    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.044706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.044673    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"]
Oct 30 22:26:00.045119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045094    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:00.045119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045107    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:00.045192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045126    4410 cpu_manager.go:414] "containerMap: removing container" podUID="7aab9c77-301f-4949-a117-c7f2f3cdbbb7" containerName="revoke-keys"
Oct 30 22:26:00.045192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045132    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aab9c77-301f-4949-a117-c7f2f3cdbbb7" containerName="revoke-keys"
Oct 30 22:26:00.045192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045146    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d05497d8-b999-4ba1-9449-79676d5d3da1" containerName="revoke-keys"
Oct 30 22:26:00.045192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045151    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05497d8-b999-4ba1-9449-79676d5d3da1" containerName="revoke-keys"
Oct 30 22:26:00.045318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045226    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05497d8-b999-4ba1-9449-79676d5d3da1" containerName="revoke-keys"
Oct 30 22:26:00.045318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045233    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:00.045318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.045243    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aab9c77-301f-4949-a117-c7f2f3cdbbb7" containerName="revoke-keys"
Oct 30 22:26:00.048369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.048353    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.100024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.099998    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:00.131333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.131308    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.131481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.131383    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.131743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.131714    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdms\" (UniqueName: \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.149367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.149345    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:26:00.149367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.149353    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:00.149518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.149489    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:00.151857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.151837    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"]
Oct 30 22:26:00.233430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.233404    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.233555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.233504    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.233634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.233589    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdms\" (UniqueName: \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.234151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.234132    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.235595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.235575    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.290430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.290406    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdms\" (UniqueName: \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\") pod \"maas-api-revoke-sub-cded56eb-qmjmq\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.360876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.360825    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:00.732550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:00.732524    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"]
Oct 30 22:26:00.735032 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:00.734992    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9603c8_a68d_4528_927d_02e3c9e54f07.slice/crio-5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46 WatchSource:0}: Error finding container 5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46: Status 404 returned error can't find the container with id 5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46
Oct 30 22:26:01.350180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:01.350137    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq" event={"ID":"9f9603c8-a68d-4528-927d-02e3c9e54f07","Type":"ContainerStarted","Data":"c6eb81a2cd20dc54a394cdd718fe0bb85195c76e75e21cd8b897b4165a85d83f"}
Oct 30 22:26:01.350558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:01.350198    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq" event={"ID":"9f9603c8-a68d-4528-927d-02e3c9e54f07","Type":"ContainerStarted","Data":"5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46"}
Oct 30 22:26:01.458812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:01.458783    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.055764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.055724    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.210660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.210627    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.358660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.358571    4410 generic.go:356] "Generic (PLEG): container finished" podID="9f9603c8-a68d-4528-927d-02e3c9e54f07" containerID="c6eb81a2cd20dc54a394cdd718fe0bb85195c76e75e21cd8b897b4165a85d83f" exitCode=0
Oct 30 22:26:02.359054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.358652    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq" event={"ID":"9f9603c8-a68d-4528-927d-02e3c9e54f07","Type":"ContainerDied","Data":"c6eb81a2cd20dc54a394cdd718fe0bb85195c76e75e21cd8b897b4165a85d83f"}
Oct 30 22:26:02.605108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.605074    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:26:02.605529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.605517    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:02.605581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.605530    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:02.605621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.605612    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba18dc36-4920-49b2-a111-498e08aa3344" containerName="istio-proxy"
Oct 30 22:26:02.608833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.608767    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.655274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.655237    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6f5f\" (UniqueName: \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.655274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.655284    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.662856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.662830    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5b1e39\""
Oct 30 22:26:02.663236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.663195    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5b1e39-dockercfg-zld96\""
Oct 30 22:26:02.666049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.666021    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:26:02.756018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.755977    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6f5f\" (UniqueName: \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.756018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.756023    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.756722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.756696    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.812516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.812482    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6f5f\" (UniqueName: \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\") pod \"payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:02.922574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:02.922541    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:03.296118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.296094    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:26:03.298135 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:03.298106    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccf9561_271c_449f_83a2_16d1ad811ad6.slice/crio-db6b0253383d1531a5be623ebc5e735548a139f3272b0cfa5de013e1e6289790 WatchSource:0}: Error finding container db6b0253383d1531a5be623ebc5e735548a139f3272b0cfa5de013e1e6289790: Status 404 returned error can't find the container with id db6b0253383d1531a5be623ebc5e735548a139f3272b0cfa5de013e1e6289790
Oct 30 22:26:03.366714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.366680    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" event={"ID":"8ccf9561-271c-449f-83a2-16d1ad811ad6","Type":"ContainerStarted","Data":"db6b0253383d1531a5be623ebc5e735548a139f3272b0cfa5de013e1e6289790"}
Oct 30 22:26:03.488898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.488872    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:03.564071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.564014    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\" (UniqueName: \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\") pod \"9f9603c8-a68d-4528-927d-02e3c9e54f07\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") "
Oct 30 22:26:03.564175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.564074    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\") pod \"9f9603c8-a68d-4528-927d-02e3c9e54f07\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") "
Oct 30 22:26:03.564239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.564223    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\") pod \"9f9603c8-a68d-4528-927d-02e3c9e54f07\" (UID: \"9f9603c8-a68d-4528-927d-02e3c9e54f07\") "
Oct 30 22:26:03.564467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.564438    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca" pod "9f9603c8-a68d-4528-927d-02e3c9e54f07" (UID: "9f9603c8-a68d-4528-927d-02e3c9e54f07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:03.564602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.564589    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.566117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.566095    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca" pod "9f9603c8-a68d-4528-927d-02e3c9e54f07" (UID: "9f9603c8-a68d-4528-927d-02e3c9e54f07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:03.566208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.566160    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms" pod "9f9603c8-a68d-4528-927d-02e3c9e54f07" (UID: "9f9603c8-a68d-4528-927d-02e3c9e54f07"). InnerVolumeSpecName "kube-api-access-cxdms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:03.665959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.665931    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9603c8-a68d-4528-927d-02e3c9e54f07-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.665959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:03.665954    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdms\" (UniqueName: \"kubernetes.io/projected/9f9603c8-a68d-4528-927d-02e3c9e54f07-kube-api-access-cxdms\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:04.373337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.373299    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" event={"ID":"8ccf9561-271c-449f-83a2-16d1ad811ad6","Type":"ContainerStarted","Data":"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"}
Oct 30 22:26:04.373727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.373365    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:04.374643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.374622    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"
Oct 30 22:26:04.374739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.374674    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq" event={"ID":"9f9603c8-a68d-4528-927d-02e3c9e54f07","Type":"ContainerDied","Data":"5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46"}
Oct 30 22:26:04.374739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.374715    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5efa21119a670aef746cf72e0e4ea6eb115c44369728c9df541d88152a9bed46"
Oct 30 22:26:04.485891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:04.485841    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" podStartSLOduration=2.485823968 podStartE2EDuration="2.485823968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:04.485623892 +0000 UTC m=+2617.574361697" watchObservedRunningTime="2026-10-30 22:26:04.485823968 +0000 UTC m=+2617.574561768"
Oct 30 22:26:06.490796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:06.490738    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:10.602689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.602652    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:26:10.603119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.603097    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9f9603c8-a68d-4528-927d-02e3c9e54f07" containerName="revoke-keys"
Oct 30 22:26:10.603119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.603110    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9603c8-a68d-4528-927d-02e3c9e54f07" containerName="revoke-keys"
Oct 30 22:26:10.603195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.603177    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9603c8-a68d-4528-927d-02e3c9e54f07" containerName="revoke-keys"
Oct 30 22:26:10.606451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.606434    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.658476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.658451    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\""
Oct 30 22:26:10.719289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.719262    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:26:10.735445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735414    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735461    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735595    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735640    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735725    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735797    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924xx\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735851    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.735999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.735979    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.736100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.736080    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.836634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836599    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.836819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836643    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.836819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836685    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.836819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836717    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924xx\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.836819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836762    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836850    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836928    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.836996    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837003    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837029    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837409    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837423    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837520    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.837811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.837763    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.839506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.839482    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.839612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.839595    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.885488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.885426    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"]
Oct 30 22:26:10.890305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.890289    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:10.893052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.893022    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.893157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.893114    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-924xx\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.918903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.918886    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:10.948323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.948282    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"]
Oct 30 22:26:10.948516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.948501    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:10.948560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.948546    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:10.948612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.948602    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:10.948678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:10.948669    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:26:11.039595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.039559    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.039802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.039603    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.039802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.039678    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4qn\" (UniqueName: \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.141154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.141119    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.141346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.141162    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.141346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.141186    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4qn\" (UniqueName: \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.141888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.141865    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.143493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.143476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.197571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.197541    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4qn\" (UniqueName: \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\") pod \"maas-api-revoke-sub-b8172f7e-5kbwv\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.213418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.213390    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:11.348301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.348273    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:26:11.350207 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:11.350181    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3df4a2_a503_4563_a096_f81435961847.slice/crio-9e4d9d9426e497098585b513c7d660b526935a386cb6b1a1759b3a3d8a22dd19 WatchSource:0}: Error finding container 9e4d9d9426e497098585b513c7d660b526935a386cb6b1a1759b3a3d8a22dd19: Status 404 returned error can't find the container with id 9e4d9d9426e497098585b513c7d660b526935a386cb6b1a1759b3a3d8a22dd19
Oct 30 22:26:11.352641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.352609    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:26:11.352728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.352670    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:26:11.352728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.352720    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:26:11.415145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.415121    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" event={"ID":"2b3df4a2-a503-4563-a096-f81435961847","Type":"ContainerStarted","Data":"9e4d9d9426e497098585b513c7d660b526935a386cb6b1a1759b3a3d8a22dd19"}
Oct 30 22:26:11.431363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.431335    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:11.585860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:11.585801    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"]
Oct 30 22:26:11.589028 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:11.588998    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980fd268_584f_46ab_9b0c_ecbdc26963ce.slice/crio-295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767 WatchSource:0}: Error finding container 295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767: Status 404 returned error can't find the container with id 295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767
Oct 30 22:26:12.421849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.421805    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv" event={"ID":"980fd268-584f-46ab-9b0c-ecbdc26963ce","Type":"ContainerStarted","Data":"95b108d801eb2ab7394565041cb989dc96658716f9fda26f4982e52d8335b078"}
Oct 30 22:26:12.422282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.421874    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv" event={"ID":"980fd268-584f-46ab-9b0c-ecbdc26963ce","Type":"ContainerStarted","Data":"295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767"}
Oct 30 22:26:12.423516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.423486    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" event={"ID":"2b3df4a2-a503-4563-a096-f81435961847","Type":"ContainerStarted","Data":"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"}
Oct 30 22:26:12.536307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.536243    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv" podStartSLOduration=2.536228631 podStartE2EDuration="2.536228631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:12.535272113 +0000 UTC m=+2625.624009915" watchObservedRunningTime="2026-10-30 22:26:12.536228631 +0000 UTC m=+2625.624966434"
Oct 30 22:26:12.674382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.674254    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podStartSLOduration=2.674234264 podStartE2EDuration="2.674234264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:12.671426276 +0000 UTC m=+2625.760164080" watchObservedRunningTime="2026-10-30 22:26:12.674234264 +0000 UTC m=+2625.762972070"
Oct 30 22:26:12.920088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.920054    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:12.921608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.921580    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:12.921759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:12.921638    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:13.430782 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:13.430748    4410 generic.go:356] "Generic (PLEG): container finished" podID="980fd268-584f-46ab-9b0c-ecbdc26963ce" containerID="95b108d801eb2ab7394565041cb989dc96658716f9fda26f4982e52d8335b078" exitCode=0
Oct 30 22:26:13.431255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:13.430832    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv" event={"ID":"980fd268-584f-46ab-9b0c-ecbdc26963ce","Type":"ContainerDied","Data":"95b108d801eb2ab7394565041cb989dc96658716f9fda26f4982e52d8335b078"}
Oct 30 22:26:13.568797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:13.567834    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.919948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:13.919907    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:13.920131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:13.919969    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:14.631478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.631452    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:14.803209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.803135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\" (UniqueName: \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\") pod \"980fd268-584f-46ab-9b0c-ecbdc26963ce\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") "
Oct 30 22:26:14.803364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.803334    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\") pod \"980fd268-584f-46ab-9b0c-ecbdc26963ce\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") "
Oct 30 22:26:14.803415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.803363    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\") pod \"980fd268-584f-46ab-9b0c-ecbdc26963ce\" (UID: \"980fd268-584f-46ab-9b0c-ecbdc26963ce\") "
Oct 30 22:26:14.803816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.803769    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca" pod "980fd268-584f-46ab-9b0c-ecbdc26963ce" (UID: "980fd268-584f-46ab-9b0c-ecbdc26963ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:14.805303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.805279    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn" pod "980fd268-584f-46ab-9b0c-ecbdc26963ce" (UID: "980fd268-584f-46ab-9b0c-ecbdc26963ce"). InnerVolumeSpecName "kube-api-access-lj4qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:14.805396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.805306    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca" pod "980fd268-584f-46ab-9b0c-ecbdc26963ce" (UID: "980fd268-584f-46ab-9b0c-ecbdc26963ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:14.904415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.904388    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:14.904415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.904415    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980fd268-584f-46ab-9b0c-ecbdc26963ce-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:14.904594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.904424    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4qn\" (UniqueName: \"kubernetes.io/projected/980fd268-584f-46ab-9b0c-ecbdc26963ce-kube-api-access-lj4qn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:14.920274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.920247    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:14.920386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:14.920294    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:15.384362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.384333    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:26:15.442837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.442804    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv" event={"ID":"980fd268-584f-46ab-9b0c-ecbdc26963ce","Type":"ContainerDied","Data":"295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767"}
Oct 30 22:26:15.442837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.442828    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"
Oct 30 22:26:15.443052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.442861    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295607bbff98def910eb78a59f625471108a202089ee775fa39a666d621ef767"
Oct 30 22:26:15.919946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.919909    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:15.920352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:15.919959    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:16.364696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:16.364605    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:16.920211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:16.920172    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:16.920660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:16.920227    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:17.919623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:17.919590    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 22:26:17.919866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:17.919641    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 22:26:18.923732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:18.923703    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:18.924172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:18.923985    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:18.924607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:18.924590    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:26:20.522944 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:26:20.522902    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:26:21.257022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:21.256989    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:26.497412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:26.497372    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:27.274023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.273987    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"]
Oct 30 22:26:27.274528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.274507    4410 cpu_manager.go:414] "containerMap: removing container" podUID="980fd268-584f-46ab-9b0c-ecbdc26963ce" containerName="revoke-keys"
Oct 30 22:26:27.274528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.274528    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980fd268-584f-46ab-9b0c-ecbdc26963ce" containerName="revoke-keys"
Oct 30 22:26:27.274708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.274630    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="980fd268-584f-46ab-9b0c-ecbdc26963ce" containerName="revoke-keys"
Oct 30 22:26:27.277864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.277844    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.335205 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.335179    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8wl\" (UniqueName: \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.335397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.335221    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.335446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.335426    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.339537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.339505    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"]
Oct 30 22:26:27.385483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.385464    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:27.385483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.385466    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:27.385483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.385486    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:26:27.385654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.385536    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:27.436249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.436225    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.436371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.436262    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8wl\" (UniqueName: \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.436371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.436280    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.436952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.436933    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.438541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.438520    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.494357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.494332    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8wl\" (UniqueName: \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\") pod \"maas-api-revoke-sub-1f1ea7d3-85fpg\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.591623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.591553    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:27.965111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:27.965086    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"]
Oct 30 22:26:27.966337 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:27.966307    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f5a9d0_fa76_4fff_801f_ba633ef4e7e0.slice/crio-666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5 WatchSource:0}: Error finding container 666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5: Status 404 returned error can't find the container with id 666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5
Oct 30 22:26:28.483502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:28.483472    4410 scope.go:122] "RemoveContainer" containerID="d469d985cccc32ca22645eb3ac9e61a7f3afcb7f62652ec2aa9bb69a1a8607ca"
Oct 30 22:26:28.544641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:28.544607    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg" event={"ID":"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0","Type":"ContainerStarted","Data":"f1c126ff5533337f202ae6ac5e83ac061d29be4abea0f4b8a2af423d29a54c7f"}
Oct 30 22:26:28.544725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:28.544668    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg" event={"ID":"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0","Type":"ContainerStarted","Data":"666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5"}
Oct 30 22:26:28.639362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:28.639314    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:28.659342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:28.659292    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg" podStartSLOduration=1.6592768 podStartE2EDuration="1.6592768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:28.657589775 +0000 UTC m=+2641.746327589" watchObservedRunningTime="2026-10-30 22:26:28.6592768 +0000 UTC m=+2641.748014647"
Oct 30 22:26:29.552440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:29.552408    4410 generic.go:356] "Generic (PLEG): container finished" podID="84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" containerID="f1c126ff5533337f202ae6ac5e83ac061d29be4abea0f4b8a2af423d29a54c7f" exitCode=0
Oct 30 22:26:29.552620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:29.552501    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg" event={"ID":"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0","Type":"ContainerDied","Data":"f1c126ff5533337f202ae6ac5e83ac061d29be4abea0f4b8a2af423d29a54c7f"}
Oct 30 22:26:30.685221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.685200    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:30.769061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.769034    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\") pod \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") "
Oct 30 22:26:30.769187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.769161    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\") pod \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") "
Oct 30 22:26:30.769243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.769225    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\" (UniqueName: \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\") pod \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\" (UID: \"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0\") "
Oct 30 22:26:30.769522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.769497    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca" pod "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" (UID: "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:30.770900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.770876    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca" pod "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" (UID: "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:30.771100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.771079    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl" pod "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" (UID: "84f5a9d0-fa76-4fff-801f-ba633ef4e7e0"). InnerVolumeSpecName "kube-api-access-pg8wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:30.870883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.870822    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:30.870883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.870845    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8wl\" (UniqueName: \"kubernetes.io/projected/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-kube-api-access-pg8wl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:30.870883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:30.870855    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:31.530574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:31.530538    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:31.572752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:31.572714    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"
Oct 30 22:26:31.572968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:31.572717    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg" event={"ID":"84f5a9d0-fa76-4fff-801f-ba633ef4e7e0","Type":"ContainerDied","Data":"666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5"}
Oct 30 22:26:31.572968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:31.572906    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666dfde3f540bb11cd97b244c2759353217945b31d9e3adf9aa2ba9bfd811ea5"
Oct 30 22:26:35.210139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:35.210100    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:35.781410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:35.781378    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.316521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:36.316475    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.521709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:36.521672    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.385173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:37.385133    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.599722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:37.599685    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.779469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:37.779431    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.936665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:37.936620    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"]
Oct 30 22:26:37.984159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:37.984121    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-955887-lzlfs"]
Oct 30 22:26:38.357530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:38.357491    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:39.019563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:39.019526    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aab9c77-301f-4949-a117-c7f2f3cdbbb7" path="/var/lib/kubelet/pods/7aab9c77-301f-4949-a117-c7f2f3cdbbb7/volumes"
Oct 30 22:26:40.817606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:40.817522    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.907594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:42.907560    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.063264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:43.063228    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.351889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:43.351806    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.700564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:44.700527    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.908427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:44.908386    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:45.089640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:45.089549    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:47.967784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:47.967736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.115021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.114987    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"]
Oct 30 22:26:48.115412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.115400    4410 cpu_manager.go:414] "containerMap: removing container" podUID="84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" containerName="revoke-keys"
Oct 30 22:26:48.115460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.115413    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" containerName="revoke-keys"
Oct 30 22:26:48.115508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.115499    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" containerName="revoke-keys"
Oct 30 22:26:48.119966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.119950    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.176692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.176664    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"]
Oct 30 22:26:48.222947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.222880    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:48.222947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.222903    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:48.222947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.222926    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:48.223142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.222967    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:26:48.257872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.257843    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.258013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.257928    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjjf\" (UniqueName: \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.258085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.258057    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.358764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.358738    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.358901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.358848    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjjf\" (UniqueName: \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.358901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.358892    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.359392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.359368    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.361231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.361212    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.415290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.415260    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjjf\" (UniqueName: \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\") pod \"maas-api-revoke-sub-4389fcd9-5gghr\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.432507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.432485    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:48.807497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:48.807462    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"]
Oct 30 22:26:48.808097 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:26:48.808060    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e08019_b2dc_4e45_a403_9f5c115e6013.slice/crio-5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218 WatchSource:0}: Error finding container 5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218: Status 404 returned error can't find the container with id 5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218
Oct 30 22:26:49.678157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:49.678110    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr" event={"ID":"07e08019-b2dc-4e45-a403-9f5c115e6013","Type":"ContainerStarted","Data":"0ad601ddc8b8463109a28c871c500ac88a459ae684015c9011e0d82b7ab15126"}
Oct 30 22:26:49.678554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:49.678173    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr" event={"ID":"07e08019-b2dc-4e45-a403-9f5c115e6013","Type":"ContainerStarted","Data":"5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218"}
Oct 30 22:26:49.793143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:49.793078    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr" podStartSLOduration=1.793060874 podStartE2EDuration="1.793060874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:49.79263446 +0000 UTC m=+2662.881372266" watchObservedRunningTime="2026-10-30 22:26:49.793060874 +0000 UTC m=+2662.881798678"
Oct 30 22:26:50.527789 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:26:50.527737    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:26:50.685356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:50.685322    4410 generic.go:356] "Generic (PLEG): container finished" podID="07e08019-b2dc-4e45-a403-9f5c115e6013" containerID="0ad601ddc8b8463109a28c871c500ac88a459ae684015c9011e0d82b7ab15126" exitCode=0
Oct 30 22:26:50.685827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:50.685406    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr" event={"ID":"07e08019-b2dc-4e45-a403-9f5c115e6013","Type":"ContainerDied","Data":"0ad601ddc8b8463109a28c871c500ac88a459ae684015c9011e0d82b7ab15126"}
Oct 30 22:26:51.836456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.836428    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:51.911764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.911734    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\" (UniqueName: \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\") pod \"07e08019-b2dc-4e45-a403-9f5c115e6013\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") "
Oct 30 22:26:51.911978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.911892    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\") pod \"07e08019-b2dc-4e45-a403-9f5c115e6013\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") "
Oct 30 22:26:51.911978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.911927    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\") pod \"07e08019-b2dc-4e45-a403-9f5c115e6013\" (UID: \"07e08019-b2dc-4e45-a403-9f5c115e6013\") "
Oct 30 22:26:51.912296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.912272    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca" pod "07e08019-b2dc-4e45-a403-9f5c115e6013" (UID: "07e08019-b2dc-4e45-a403-9f5c115e6013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:51.913849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.913828    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf" pod "07e08019-b2dc-4e45-a403-9f5c115e6013" (UID: "07e08019-b2dc-4e45-a403-9f5c115e6013"). InnerVolumeSpecName "kube-api-access-2qjjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:51.913926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:51.913869    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca" pod "07e08019-b2dc-4e45-a403-9f5c115e6013" (UID: "07e08019-b2dc-4e45-a403-9f5c115e6013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:52.013166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.013079    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjjf\" (UniqueName: \"kubernetes.io/projected/07e08019-b2dc-4e45-a403-9f5c115e6013-kube-api-access-2qjjf\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:52.013166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.013108    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:52.013166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.013123    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e08019-b2dc-4e45-a403-9f5c115e6013-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:52.698119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.698089    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"
Oct 30 22:26:52.698119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.698104    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr" event={"ID":"07e08019-b2dc-4e45-a403-9f5c115e6013","Type":"ContainerDied","Data":"5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218"}
Oct 30 22:26:52.698452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.698152    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f94a859d2be1deddd8a68661775188fde5f4ccfe61f837065bddedc81e55218"
Oct 30 22:26:52.953224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:52.953133    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.232908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:26:58.232874    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:03.371628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:03.371592    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.584185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.584151    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"]
Oct 30 22:27:04.584613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.584600    4410 cpu_manager.go:414] "containerMap: removing container" podUID="07e08019-b2dc-4e45-a403-9f5c115e6013" containerName="revoke-keys"
Oct 30 22:27:04.584657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.584614    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e08019-b2dc-4e45-a403-9f5c115e6013" containerName="revoke-keys"
Oct 30 22:27:04.584706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.584697    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e08019-b2dc-4e45-a403-9f5c115e6013" containerName="revoke-keys"
Oct 30 22:27:04.588042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.588025    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.645885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.645855    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"]
Oct 30 22:27:04.661879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.661852    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtf7\" (UniqueName: \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.662030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.662001    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.662080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.662070    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.691842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.691815    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:27:04.691948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.691874    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:04.691948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.691881    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:04.691948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.691927    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:04.762682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.762651    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.762856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.762698    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.762856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.762819    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtf7\" (UniqueName: \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.763375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.763350    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.765071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.765048    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.824194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.824171    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtf7\" (UniqueName: \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\") pod \"maas-api-revoke-sub-c237c82d-dqxbt\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:04.900454 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:04.900428    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:05.277386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:05.277362    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"]
Oct 30 22:27:05.279320 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:05.279291    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b177c16_c678_4d1e_9984_68ff3c2af978.slice/crio-3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d WatchSource:0}: Error finding container 3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d: Status 404 returned error can't find the container with id 3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d
Oct 30 22:27:05.776526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:05.776489    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt" event={"ID":"6b177c16-c678-4d1e-9984-68ff3c2af978","Type":"ContainerStarted","Data":"baa0acac1f38bf4efd277a05b8ef415c8a14a1ad57b803c938dd0a291cc37e90"}
Oct 30 22:27:05.776905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:05.776553    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt" event={"ID":"6b177c16-c678-4d1e-9984-68ff3c2af978","Type":"ContainerStarted","Data":"3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d"}
Oct 30 22:27:05.897256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:05.897208    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt" podStartSLOduration=1.897194216 podStartE2EDuration="1.897194216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:05.896327207 +0000 UTC m=+2678.985065011" watchObservedRunningTime="2026-10-30 22:27:05.897194216 +0000 UTC m=+2678.985932019"
Oct 30 22:27:06.784236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:06.784201    4410 generic.go:356] "Generic (PLEG): container finished" podID="6b177c16-c678-4d1e-9984-68ff3c2af978" containerID="baa0acac1f38bf4efd277a05b8ef415c8a14a1ad57b803c938dd0a291cc37e90" exitCode=0
Oct 30 22:27:06.784654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:06.784296    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt" event={"ID":"6b177c16-c678-4d1e-9984-68ff3c2af978","Type":"ContainerDied","Data":"baa0acac1f38bf4efd277a05b8ef415c8a14a1ad57b803c938dd0a291cc37e90"}
Oct 30 22:27:07.922516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:07.922492    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:08.099011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.098941    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\" (UniqueName: \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\") pod \"6b177c16-c678-4d1e-9984-68ff3c2af978\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") "
Oct 30 22:27:08.099185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.099021    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\") pod \"6b177c16-c678-4d1e-9984-68ff3c2af978\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") "
Oct 30 22:27:08.099185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.099058    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\") pod \"6b177c16-c678-4d1e-9984-68ff3c2af978\" (UID: \"6b177c16-c678-4d1e-9984-68ff3c2af978\") "
Oct 30 22:27:08.099465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.099439    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca" pod "6b177c16-c678-4d1e-9984-68ff3c2af978" (UID: "6b177c16-c678-4d1e-9984-68ff3c2af978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:08.099539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.099512    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:08.101020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.100995    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca" pod "6b177c16-c678-4d1e-9984-68ff3c2af978" (UID: "6b177c16-c678-4d1e-9984-68ff3c2af978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:08.101142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.101120    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7" pod "6b177c16-c678-4d1e-9984-68ff3c2af978" (UID: "6b177c16-c678-4d1e-9984-68ff3c2af978"). InnerVolumeSpecName "kube-api-access-mxtf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:08.200808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.200763    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtf7\" (UniqueName: \"kubernetes.io/projected/6b177c16-c678-4d1e-9984-68ff3c2af978-kube-api-access-mxtf7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:08.200808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.200806    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b177c16-c678-4d1e-9984-68ff3c2af978-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:08.439145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.439098    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:08.797602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.797511    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"
Oct 30 22:27:08.797602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.797537    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt" event={"ID":"6b177c16-c678-4d1e-9984-68ff3c2af978","Type":"ContainerDied","Data":"3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d"}
Oct 30 22:27:08.797602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:08.797584    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb57ec0fc646eae859a1b3a3ab5c0e68f0ee195768405975bc11dd01aef1e0d"
Oct 30 22:27:13.554500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:13.554468    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.636258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:16.636218    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.333951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:17.333918    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.599357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:17.599272    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.499262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:18.499224    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:20.287484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:20.287446    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:20.450907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:20.450870    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:20.522387 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:27:20.522355    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:27:21.773079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:21.773039    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:21.957712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:21.957666    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:22.144766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:22.144729    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.574598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:23.574564    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:25.101722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.101688    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:25.306464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.306429    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:25.530947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.530913    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"]
Oct 30 22:27:25.531352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.531340    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6b177c16-c678-4d1e-9984-68ff3c2af978" containerName="revoke-keys"
Oct 30 22:27:25.531399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.531353    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b177c16-c678-4d1e-9984-68ff3c2af978" containerName="revoke-keys"
Oct 30 22:27:25.531440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.531430    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b177c16-c678-4d1e-9984-68ff3c2af978" containerName="revoke-keys"
Oct 30 22:27:25.534918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.534900    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.579391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.579364    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.579545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.579526    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkrj\" (UniqueName: \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.579588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.579559    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.589185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.589164    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:25.589277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.589212    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:27:25.592256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.592241    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:25.649011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.648976    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"]
Oct 30 22:27:25.652259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.652030    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:27:25.658534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.658510    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.680692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.680667    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.680835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.680793    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkrj\" (UniqueName: \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.680835 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.680826    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.683157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.683135    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.693519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.693499    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:25.696317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.696295    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:27:25.700313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.700295    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.701629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.701607    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.715890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.715861    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:27:25.741649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.741516    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkrj\" (UniqueName: \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\") pod \"maas-api-revoke-sub-3d4050ee-c8fws\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.766875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.766290    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:27:25.781812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781736    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.781812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781784    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.781983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781851    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.781983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781886    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.781983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781911    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.781997    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.782148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782131    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782173    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.782293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782227    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782283    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.782415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782310    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvlj\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782387    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782458    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.782586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782553    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782578    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq765\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.782687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782603    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782695    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.782767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.782726    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.848944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.848912    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:25.884086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884053    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.884233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884112    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.884233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884148    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.884233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884175    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvlj\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.884487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884463    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.884589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884569    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.884695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884678    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.884901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884686    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.885004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884897    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.885004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884986    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.885117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.885093    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.885190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.884881    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.885320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.885299    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq765\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.885454 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.885438    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886120    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886056    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886162    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886258    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886297    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886325    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886371    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886407    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886412    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886527    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.886611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886605    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886681    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.886985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.886960    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.887248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.887225    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.890904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.890803    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.890904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.890861    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.890904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.890882    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.893002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.892979    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.941710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.941663    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvlj\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.942608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.942533    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:25.942851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.942830    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq765\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.944723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.944701    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:25.972074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:25.972052    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:26.013734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.013709    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:26.232326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.232297    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"]
Oct 30 22:27:26.234317 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:26.234287    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eeda028_d249_4b4f_bd20_6a4e9ba98610.slice/crio-905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30 WatchSource:0}: Error finding container 905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30: Status 404 returned error can't find the container with id 905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30
Oct 30 22:27:26.356787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.356749    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:27:26.358618 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:26.358591    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960c1ab3_72c2_45ed_9a7e_edc36297c59a.slice/crio-6fcfd40008892461a0a18e30a8a8efb51c7308d55ac1b252a74e3089fe28f6b9 WatchSource:0}: Error finding container 6fcfd40008892461a0a18e30a8a8efb51c7308d55ac1b252a74e3089fe28f6b9: Status 404 returned error can't find the container with id 6fcfd40008892461a0a18e30a8a8efb51c7308d55ac1b252a74e3089fe28f6b9
Oct 30 22:27:26.361012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.360980    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.361101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.361041    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.361101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.361069    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.423517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.423493    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:27:26.426324 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:26.426302    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb225fa_2b45_4988_9d05_b83a1fb3ba2e.slice/crio-95b7bd42a5f0dfe14d3f505a46857f3de479195be3ee9c26a4be7629429f3fae WatchSource:0}: Error finding container 95b7bd42a5f0dfe14d3f505a46857f3de479195be3ee9c26a4be7629429f3fae: Status 404 returned error can't find the container with id 95b7bd42a5f0dfe14d3f505a46857f3de479195be3ee9c26a4be7629429f3fae
Oct 30 22:27:26.428653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.428625    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.428723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.428690    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.428723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.428719    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:27:26.914799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.914725    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" event={"ID":"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e","Type":"ContainerStarted","Data":"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"}
Oct 30 22:27:26.915115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.915090    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" event={"ID":"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e","Type":"ContainerStarted","Data":"95b7bd42a5f0dfe14d3f505a46857f3de479195be3ee9c26a4be7629429f3fae"}
Oct 30 22:27:26.918203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.918169    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws" event={"ID":"6eeda028-d249-4b4f-bd20-6a4e9ba98610","Type":"ContainerStarted","Data":"e1bf938985439c5d358694b24924f0ba4c0910db6e32df5e7ac6889dfc72ccb2"}
Oct 30 22:27:26.918447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.918421    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws" event={"ID":"6eeda028-d249-4b4f-bd20-6a4e9ba98610","Type":"ContainerStarted","Data":"905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30"}
Oct 30 22:27:26.921352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.921326    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" event={"ID":"960c1ab3-72c2-45ed-9a7e-edc36297c59a","Type":"ContainerStarted","Data":"0ac224a10cff11f7cbcc99c8bb499a964325f271ec1f07265869826c46df1810"}
Oct 30 22:27:26.921537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.921520    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" event={"ID":"960c1ab3-72c2-45ed-9a7e-edc36297c59a","Type":"ContainerStarted","Data":"6fcfd40008892461a0a18e30a8a8efb51c7308d55ac1b252a74e3089fe28f6b9"}
Oct 30 22:27:26.977991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:26.977958    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:27.017139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.016956    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:27.017139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.017012    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:27.021485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.021438    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:27.053797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.045519    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podStartSLOduration=2.045502936 podStartE2EDuration="2.045502936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.041129159 +0000 UTC m=+2700.129866952" watchObservedRunningTime="2026-10-30 22:27:27.045502936 +0000 UTC m=+2700.134240740"
Oct 30 22:27:27.323824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.323758    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws" podStartSLOduration=2.323745992 podStartE2EDuration="2.323745992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.32184934 +0000 UTC m=+2700.410587140" watchObservedRunningTime="2026-10-30 22:27:27.323745992 +0000 UTC m=+2700.412483796"
Oct 30 22:27:27.324419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.324389    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podStartSLOduration=2.324380324 podStartE2EDuration="2.324380324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.196819469 +0000 UTC m=+2700.285557273" watchObservedRunningTime="2026-10-30 22:27:27.324380324 +0000 UTC m=+2700.413118126"
Oct 30 22:27:27.928905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.928865    4410 generic.go:356] "Generic (PLEG): container finished" podID="6eeda028-d249-4b4f-bd20-6a4e9ba98610" containerID="e1bf938985439c5d358694b24924f0ba4c0910db6e32df5e7ac6889dfc72ccb2" exitCode=0
Oct 30 22:27:27.929874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.929850    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws" event={"ID":"6eeda028-d249-4b4f-bd20-6a4e9ba98610","Type":"ContainerDied","Data":"e1bf938985439c5d358694b24924f0ba4c0910db6e32df5e7ac6889dfc72ccb2"}
Oct 30 22:27:27.978798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.978676    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:27.978798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:27.978739    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:28.014691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.014655    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:28.014880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.014708    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:28.510916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.510885    4410 scope.go:122] "RemoveContainer" containerID="08cc3f592b889f8bfe0a3e692533e1977dfa97bdbc8174072d7ff42154876b5f"
Oct 30 22:27:28.566431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.566398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:28.987798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.977367    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:28.987798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.977946    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:28.987798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:28.977994    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:29.033725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.028694    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:29.033725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.028757    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:29.319798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.319660    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:29.401817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.400251    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\" (UniqueName: \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\") pod \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") "
Oct 30 22:27:29.401817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.400386    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\") pod \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") "
Oct 30 22:27:29.401817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.400475    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\") pod \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\" (UID: \"6eeda028-d249-4b4f-bd20-6a4e9ba98610\") "
Oct 30 22:27:29.414806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.413232    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca" pod "6eeda028-d249-4b4f-bd20-6a4e9ba98610" (UID: "6eeda028-d249-4b4f-bd20-6a4e9ba98610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:29.414976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.414954    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca" pod "6eeda028-d249-4b4f-bd20-6a4e9ba98610" (UID: "6eeda028-d249-4b4f-bd20-6a4e9ba98610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:29.420273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.420237    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj" pod "6eeda028-d249-4b4f-bd20-6a4e9ba98610" (UID: "6eeda028-d249-4b4f-bd20-6a4e9ba98610"). InnerVolumeSpecName "kube-api-access-jtkrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:29.523194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.522701    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:29.523194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.522747    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkrj\" (UniqueName: \"kubernetes.io/projected/6eeda028-d249-4b4f-bd20-6a4e9ba98610-kube-api-access-jtkrj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:29.523194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.522765    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eeda028-d249-4b4f-bd20-6a4e9ba98610-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:29.942556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.942520    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws" event={"ID":"6eeda028-d249-4b4f-bd20-6a4e9ba98610","Type":"ContainerDied","Data":"905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30"}
Oct 30 22:27:29.942715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.942573    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="905579ac2f0b92c366ea6dfaa9e6dba8608fd1b3b1d3dffc87513f8616e14b30"
Oct 30 22:27:29.942715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.942623    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"
Oct 30 22:27:29.976799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.976159    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:29.976799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:29.976223    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:30.014869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:30.014828    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:30.015032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:30.014890    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:30.973794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:30.973227    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:30.973794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:30.973315    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:31.014379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:31.014341    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:31.014546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:31.014404    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:31.978797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:31.978193    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:31.978797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:31.978255    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:32.015047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:32.014998    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:32.015216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:32.015067    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:32.973281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:32.973240    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:27:32.973438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:32.973346    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 22:27:33.014369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:33.014332    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:27:33.014762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:33.014380    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:27:33.581786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:33.581734    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:34.988333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:34.988282    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:27:34.988894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:34.988347    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.057585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.057528    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:27:35.057765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.057622    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.126840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.126813    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:35.127150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.127129    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:35.127904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.127888    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:27:35.305966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.305891    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:35.306190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.306161    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:35.307406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:35.307384    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:27:38.607711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:38.607673    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.533005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:43.532973    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.744216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:43.744163    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:44.698736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:44.698699    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.307453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:45.307412    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.531707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:45.531670    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:46.671220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.671184    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"]
Oct 30 22:27:46.671657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.671644    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6eeda028-d249-4b4f-bd20-6a4e9ba98610" containerName="revoke-keys"
Oct 30 22:27:46.671703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.671658    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eeda028-d249-4b4f-bd20-6a4e9ba98610" containerName="revoke-keys"
Oct 30 22:27:46.671801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.671764    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eeda028-d249-4b4f-bd20-6a4e9ba98610" containerName="revoke-keys"
Oct 30 22:27:46.675329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.675312    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.765057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.765023    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvh4j\" (UniqueName: \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.765057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.765064    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.765312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.765286    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.777722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.777697    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:46.777722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.777711    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:46.777956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.777731    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:27:46.777956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.777742    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:46.780649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.780626    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"]
Oct 30 22:27:46.866644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.866610    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvh4j\" (UniqueName: \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.866847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.866651    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.866847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.866839    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.867500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.867480    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.869061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.869041    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.924376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.924312    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvh4j\" (UniqueName: \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\") pod \"maas-api-revoke-sub-30631cf6-r8p6n\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:46.988970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:46.988943    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:47.373393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:47.373364    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"]
Oct 30 22:27:47.374504 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:47.374479    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7084bd7_ae5d_4df9_ae6e_b51332042b92.slice/crio-8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068 WatchSource:0}: Error finding container 8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068: Status 404 returned error can't find the container with id 8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068
Oct 30 22:27:48.085207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:48.085111    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n" event={"ID":"c7084bd7-ae5d-4df9-ae6e-b51332042b92","Type":"ContainerStarted","Data":"84c15a1d03696c4f09859abebd6445e963257cd141efd8330b9c69eea2c724a4"}
Oct 30 22:27:48.085207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:48.085169    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n" event={"ID":"c7084bd7-ae5d-4df9-ae6e-b51332042b92","Type":"ContainerStarted","Data":"8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068"}
Oct 30 22:27:48.205220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:48.205168    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n" podStartSLOduration=2.205155269 podStartE2EDuration="2.205155269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:48.204034039 +0000 UTC m=+2721.292771843" watchObservedRunningTime="2026-10-30 22:27:48.205155269 +0000 UTC m=+2721.293893072"
Oct 30 22:27:49.056472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:49.056431    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:49.092362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:49.092331    4410 generic.go:356] "Generic (PLEG): container finished" podID="c7084bd7-ae5d-4df9-ae6e-b51332042b92" containerID="84c15a1d03696c4f09859abebd6445e963257cd141efd8330b9c69eea2c724a4" exitCode=0
Oct 30 22:27:49.092728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:49.092386    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n" event={"ID":"c7084bd7-ae5d-4df9-ae6e-b51332042b92","Type":"ContainerDied","Data":"84c15a1d03696c4f09859abebd6445e963257cd141efd8330b9c69eea2c724a4"}
Oct 30 22:27:49.263215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:49.263177    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:49.439989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:49.439947    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.245061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.245036    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:50.405021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.404995    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\" (UniqueName: \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\") pod \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") "
Oct 30 22:27:50.405199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.405181    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\") pod \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") "
Oct 30 22:27:50.405249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.405205    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\") pod \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\" (UID: \"c7084bd7-ae5d-4df9-ae6e-b51332042b92\") "
Oct 30 22:27:50.405574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.405540    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca" pod "c7084bd7-ae5d-4df9-ae6e-b51332042b92" (UID: "c7084bd7-ae5d-4df9-ae6e-b51332042b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.405719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.405703    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.407089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.407065    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j" pod "c7084bd7-ae5d-4df9-ae6e-b51332042b92" (UID: "c7084bd7-ae5d-4df9-ae6e-b51332042b92"). InnerVolumeSpecName "kube-api-access-hvh4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.407197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.407132    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca" pod "c7084bd7-ae5d-4df9-ae6e-b51332042b92" (UID: "c7084bd7-ae5d-4df9-ae6e-b51332042b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:50.507179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.507144    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvh4j\" (UniqueName: \"kubernetes.io/projected/c7084bd7-ae5d-4df9-ae6e-b51332042b92-kube-api-access-hvh4j\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.507179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:50.507172    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7084bd7-ae5d-4df9-ae6e-b51332042b92-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.521281 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:27:50.521245    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:27:51.105691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:51.105655    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n" event={"ID":"c7084bd7-ae5d-4df9-ae6e-b51332042b92","Type":"ContainerDied","Data":"8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068"}
Oct 30 22:27:51.105691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:51.105686    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"
Oct 30 22:27:51.105978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:51.105701    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8715a0fe9053b35edbd24d96eb8d300a4906f06ced42b946e410e1f1bba1a068"
Oct 30 22:27:52.335268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:52.335227    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:52.547752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:52.547702    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:54.428172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:54.428139    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:54.632521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:54.632484    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.126140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.126101    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"]
Oct 30 22:27:56.126758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.126737    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c7084bd7-ae5d-4df9-ae6e-b51332042b92" containerName="revoke-keys"
Oct 30 22:27:56.126897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.126760    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7084bd7-ae5d-4df9-ae6e-b51332042b92" containerName="revoke-keys"
Oct 30 22:27:56.126958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.126922    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7084bd7-ae5d-4df9-ae6e-b51332042b92" containerName="revoke-keys"
Oct 30 22:27:56.130436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.130414    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.184795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.184701    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:56.233919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.233888    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:56.234031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.233949    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:56.234077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.234047    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:27:56.237023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.237004    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"]
Oct 30 22:27:56.271075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.271052    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.271164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.271105    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.271216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.271175    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbgz\" (UniqueName: \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.372115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.372075    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.372415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.372393    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.372541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.372527    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbgz\" (UniqueName: \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.373184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.373155    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.392755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.392725    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.430619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.430587    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbgz\" (UniqueName: \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.444335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.444297    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:56.835522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:56.835491    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"]
Oct 30 22:27:56.839932 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:27:56.839907    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248c2b0b_8106_454b_984a_ebdee0855f72.slice/crio-1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d WatchSource:0}: Error finding container 1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d: Status 404 returned error can't find the container with id 1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d
Oct 30 22:27:57.143711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:57.143677    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg" event={"ID":"248c2b0b-8106-454b-984a-ebdee0855f72","Type":"ContainerStarted","Data":"1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d"}
Oct 30 22:27:58.158813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:58.158762    4410 generic.go:356] "Generic (PLEG): container finished" podID="248c2b0b-8106-454b-984a-ebdee0855f72" containerID="50849ef1cdd25433e39e00960fcc8d44485cc2b65e9fd3122ec95de003895906" exitCode=0
Oct 30 22:27:58.159207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:58.158846    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg" event={"ID":"248c2b0b-8106-454b-984a-ebdee0855f72","Type":"ContainerDied","Data":"50849ef1cdd25433e39e00960fcc8d44485cc2b65e9fd3122ec95de003895906"}
Oct 30 22:27:59.305109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.305078    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:27:59.333793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.333746    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\") pod \"248c2b0b-8106-454b-984a-ebdee0855f72\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") "
Oct 30 22:27:59.333926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.333811    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\" (UniqueName: \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\") pod \"248c2b0b-8106-454b-984a-ebdee0855f72\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") "
Oct 30 22:27:59.333926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.333894    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\") pod \"248c2b0b-8106-454b-984a-ebdee0855f72\" (UID: \"248c2b0b-8106-454b-984a-ebdee0855f72\") "
Oct 30 22:27:59.334095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.334073    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca" pod "248c2b0b-8106-454b-984a-ebdee0855f72" (UID: "248c2b0b-8106-454b-984a-ebdee0855f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:59.334310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.334294    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.335955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.335931    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca" pod "248c2b0b-8106-454b-984a-ebdee0855f72" (UID: "248c2b0b-8106-454b-984a-ebdee0855f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:59.336058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.336034    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz" pod "248c2b0b-8106-454b-984a-ebdee0855f72" (UID: "248c2b0b-8106-454b-984a-ebdee0855f72"). InnerVolumeSpecName "kube-api-access-rvbgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:59.435957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.435891    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248c2b0b-8106-454b-984a-ebdee0855f72-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.435957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.435914    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbgz\" (UniqueName: \"kubernetes.io/projected/248c2b0b-8106-454b-984a-ebdee0855f72-kube-api-access-rvbgz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.726417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:27:59.726332    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:00.171516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:00.171489    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"
Oct 30 22:28:00.171747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:00.171519    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg" event={"ID":"248c2b0b-8106-454b-984a-ebdee0855f72","Type":"ContainerDied","Data":"1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d"}
Oct 30 22:28:00.171747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:00.171565    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca83f6c50422e5f78d070761e3bc16f2da2b5e1bf51ca16b9a67ddc57ac733d"
Oct 30 22:28:02.053745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:02.053702    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"]
Oct 30 22:28:02.102600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:02.102563    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9022fad-brg6k"]
Oct 30 22:28:03.019115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:03.019079    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbc7224-d1b3-4bfd-a735-3e89a8e47147" path="/var/lib/kubelet/pods/9fbc7224-d1b3-4bfd-a735-3e89a8e47147/volumes"
Oct 30 22:28:04.620613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:04.620580    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:05.048358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:05.048325    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:07.653332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:07.653293    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:28:08.090389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:08.090297    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:08.252696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:08.252661    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.142512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:09.142477    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.307801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:09.307760    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.080321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:10.080247    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.654634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.654565    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" podUID="1c5b463d-d07d-43c0-8018-25349551c737" containerName="payload-processing" containerID="cri-o://6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076" gracePeriod=25
Oct 30 22:28:12.908793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.908720    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:28:12.979845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.979819    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\" (UniqueName: \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\") pod \"1c5b463d-d07d-43c0-8018-25349551c737\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") "
Oct 30 22:28:12.979991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.979980    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\") pod \"1c5b463d-d07d-43c0-8018-25349551c737\" (UID: \"1c5b463d-d07d-43c0-8018-25349551c737\") "
Oct 30 22:28:12.980311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.980286    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume" pod "1c5b463d-d07d-43c0-8018-25349551c737" (UID: "1c5b463d-d07d-43c0-8018-25349551c737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:12.981727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:12.981707    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55" pod "1c5b463d-d07d-43c0-8018-25349551c737" (UID: "1c5b463d-d07d-43c0-8018-25349551c737"). InnerVolumeSpecName "kube-api-access-4xp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:13.080957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.080925    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b463d-d07d-43c0-8018-25349551c737-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:13.080957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.080954    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xp55\" (UniqueName: \"kubernetes.io/projected/1c5b463d-d07d-43c0-8018-25349551c737-kube-api-access-4xp55\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:13.254468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.254381    4410 generic.go:356] "Generic (PLEG): container finished" podID="1c5b463d-d07d-43c0-8018-25349551c737" containerID="6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076" exitCode=0
Oct 30 22:28:13.254468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.254445    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"
Oct 30 22:28:13.254644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.254465    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" event={"ID":"1c5b463d-d07d-43c0-8018-25349551c737","Type":"ContainerDied","Data":"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"}
Oct 30 22:28:13.254644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.254544    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d" event={"ID":"1c5b463d-d07d-43c0-8018-25349551c737","Type":"ContainerDied","Data":"54008db1e64e9f563d431e2453ce0b5050daefd543180a965ce8d1a2b22fc24d"}
Oct 30 22:28:13.254644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.254568    4410 scope.go:122] "RemoveContainer" containerID="6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"
Oct 30 22:28:13.264047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.264034    4410 scope.go:122] "RemoveContainer" containerID="6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"
Oct 30 22:28:13.264346 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:28:13.264321    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076\": container with ID starting with 6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076 not found: ID does not exist" containerID="6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"
Oct 30 22:28:13.264449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.264353    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076"} err="failed to get container status \"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076\": rpc error: code = NotFound desc = could not find container \"6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076\": container with ID starting with 6a382bec5bf8e2e870a39fd136b046f288ceb21a1424fb954840b5db07e46076 not found: ID does not exist"
Oct 30 22:28:13.423406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.423378    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:28:13.471279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:13.471252    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cce50a-7c74b44fc-wqs8d"]
Oct 30 22:28:14.347917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:14.347887    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.019175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:15.019142    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5b463d-d07d-43c0-8018-25349551c737" path="/var/lib/kubelet/pods/1c5b463d-d07d-43c0-8018-25349551c737/volumes"
Oct 30 22:28:15.963254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:15.963221    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.125582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.125520    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.313888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.313808    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.471722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.471687    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.489931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.489901    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"]
Oct 30 22:28:16.538950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.538919    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-cce50a-v8jdg"]
Oct 30 22:28:16.924697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:16.924659    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:17.019102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:17.019074    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248c2b0b-8106-454b-984a-ebdee0855f72" path="/var/lib/kubelet/pods/248c2b0b-8106-454b-984a-ebdee0855f72/volumes"
Oct 30 22:28:20.522491 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:28:20.522444    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:28:21.841562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:21.841508    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.502230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:23.502197    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.680628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:23.680589    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.887528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:23.887436    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:24.356948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.356917    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"]
Oct 30 22:28:24.357354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357342    4410 cpu_manager.go:414] "containerMap: removing container" podUID="248c2b0b-8106-454b-984a-ebdee0855f72" containerName="revoke-keys"
Oct 30 22:28:24.357400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357355    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248c2b0b-8106-454b-984a-ebdee0855f72" containerName="revoke-keys"
Oct 30 22:28:24.357400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357377    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1c5b463d-d07d-43c0-8018-25349551c737" containerName="payload-processing"
Oct 30 22:28:24.357400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357384    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5b463d-d07d-43c0-8018-25349551c737" containerName="payload-processing"
Oct 30 22:28:24.357497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357475    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5b463d-d07d-43c0-8018-25349551c737" containerName="payload-processing"
Oct 30 22:28:24.357497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.357486    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="248c2b0b-8106-454b-984a-ebdee0855f72" containerName="revoke-keys"
Oct 30 22:28:24.361601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.361584    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.415811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.415769    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:24.465238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.465211    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:24.465238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.465233    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:24.465470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.465434    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:28:24.468425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.468406    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"]
Oct 30 22:28:24.521331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.521305    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzsl\" (UniqueName: \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.521649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.521342    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.521649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.521365    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.622979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.622910    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzsl\" (UniqueName: \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.622979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.622960    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.623127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.622995    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.623830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.623799    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.625366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.625338    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.656381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.656349    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"]
Oct 30 22:28:24.660204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.660185    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.682858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.682832    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzsl\" (UniqueName: \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\") pod \"maas-api-revoke-sub-44728e89-gc6z9\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.723870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.723844    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"]
Oct 30 22:28:24.825439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.825408    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.825586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.825465    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.825629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.825578    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kp6p\" (UniqueName: \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.927378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.927333    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kp6p\" (UniqueName: \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.927558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.927477    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.927558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.927525    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.928152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.928120    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.929898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.929873    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:24.974336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.974311    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:24.987547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:24.987523    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kp6p\" (UniqueName: \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\") pod \"maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:25.273396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:25.273315    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:25.354217 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:25.354190    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"]
Oct 30 22:28:25.356228 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:28:25.356204    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745987a6_cba7_49c4_95b6_bac1775035c0.slice/crio-cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1 WatchSource:0}: Error finding container cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1: Status 404 returned error can't find the container with id cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1
Oct 30 22:28:25.647104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:25.647076    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"]
Oct 30 22:28:25.648185 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:28:25.648156    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99646111_522d_4b49_b18b_8cf6665ee0a9.slice/crio-6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47 WatchSource:0}: Error finding container 6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47: Status 404 returned error can't find the container with id 6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47
Oct 30 22:28:26.354939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.354900    4410 generic.go:356] "Generic (PLEG): container finished" podID="745987a6-cba7-49c4-95b6-bac1775035c0" containerID="e94b4eeb7731b01ef4a6191f043ea85e49f77f63ce005904c8e5ddbe75d2f773" exitCode=0
Oct 30 22:28:26.355130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.355058    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9" event={"ID":"745987a6-cba7-49c4-95b6-bac1775035c0","Type":"ContainerDied","Data":"e94b4eeb7731b01ef4a6191f043ea85e49f77f63ce005904c8e5ddbe75d2f773"}
Oct 30 22:28:26.355130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.355110    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9" event={"ID":"745987a6-cba7-49c4-95b6-bac1775035c0","Type":"ContainerStarted","Data":"cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1"}
Oct 30 22:28:26.357705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.357673    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv" event={"ID":"99646111-522d-4b49-b18b-8cf6665ee0a9","Type":"ContainerStarted","Data":"c8d29a9912b28de6e49a5b870052a0399ccf2339c9bb598a0b5e47580db5654a"}
Oct 30 22:28:26.357866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.357727    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv" event={"ID":"99646111-522d-4b49-b18b-8cf6665ee0a9","Type":"ContainerStarted","Data":"6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47"}
Oct 30 22:28:26.583415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:26.583364    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv" podStartSLOduration=2.583350921 podStartE2EDuration="2.583350921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:26.582555553 +0000 UTC m=+2759.671293368" watchObservedRunningTime="2026-10-30 22:28:26.583350921 +0000 UTC m=+2759.672088724"
Oct 30 22:28:27.365232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.365194    4410 generic.go:356] "Generic (PLEG): container finished" podID="99646111-522d-4b49-b18b-8cf6665ee0a9" containerID="c8d29a9912b28de6e49a5b870052a0399ccf2339c9bb598a0b5e47580db5654a" exitCode=0
Oct 30 22:28:27.365630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.365280    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv" event={"ID":"99646111-522d-4b49-b18b-8cf6665ee0a9","Type":"ContainerDied","Data":"c8d29a9912b28de6e49a5b870052a0399ccf2339c9bb598a0b5e47580db5654a"}
Oct 30 22:28:27.467104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.467067    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:27.523626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.523600    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:27.657247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.657212    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\") pod \"745987a6-cba7-49c4-95b6-bac1775035c0\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") "
Oct 30 22:28:27.657449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.657266    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\") pod \"745987a6-cba7-49c4-95b6-bac1775035c0\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") "
Oct 30 22:28:27.657449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.657298    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\" (UniqueName: \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\") pod \"745987a6-cba7-49c4-95b6-bac1775035c0\" (UID: \"745987a6-cba7-49c4-95b6-bac1775035c0\") "
Oct 30 22:28:27.657682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.657653    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca" pod "745987a6-cba7-49c4-95b6-bac1775035c0" (UID: "745987a6-cba7-49c4-95b6-bac1775035c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:27.657839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.657821    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:27.659467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.659438    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl" pod "745987a6-cba7-49c4-95b6-bac1775035c0" (UID: "745987a6-cba7-49c4-95b6-bac1775035c0"). InnerVolumeSpecName "kube-api-access-7xzsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:27.659578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.659483    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca" pod "745987a6-cba7-49c4-95b6-bac1775035c0" (UID: "745987a6-cba7-49c4-95b6-bac1775035c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:27.758514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.758486    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745987a6-cba7-49c4-95b6-bac1775035c0-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:27.758514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:27.758516    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzsl\" (UniqueName: \"kubernetes.io/projected/745987a6-cba7-49c4-95b6-bac1775035c0-kube-api-access-7xzsl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.381234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.381202    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"
Oct 30 22:28:28.381700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.381241    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9" event={"ID":"745987a6-cba7-49c4-95b6-bac1775035c0","Type":"ContainerDied","Data":"cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1"}
Oct 30 22:28:28.381700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.381320    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc15f561e1784bfc09de999d923792dea5c9179c043623c1898c66abf4ceb2d1"
Oct 30 22:28:28.524764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.524741    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:28.535536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.535515    4410 scope.go:122] "RemoveContainer" containerID="dd38a6aba516627ce80470635eeac1f745c6e5fe530d0f02152aaed21cdac78a"
Oct 30 22:28:28.668515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.668486    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\") pod \"99646111-522d-4b49-b18b-8cf6665ee0a9\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") "
Oct 30 22:28:28.668672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.668592    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\") pod \"99646111-522d-4b49-b18b-8cf6665ee0a9\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") "
Oct 30 22:28:28.668718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.668692    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\" (UniqueName: \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\") pod \"99646111-522d-4b49-b18b-8cf6665ee0a9\" (UID: \"99646111-522d-4b49-b18b-8cf6665ee0a9\") "
Oct 30 22:28:28.668965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.668936    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca" pod "99646111-522d-4b49-b18b-8cf6665ee0a9" (UID: "99646111-522d-4b49-b18b-8cf6665ee0a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:28.669222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.669201    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.670719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.670686    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca" pod "99646111-522d-4b49-b18b-8cf6665ee0a9" (UID: "99646111-522d-4b49-b18b-8cf6665ee0a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:28.670877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.670754    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p" pod "99646111-522d-4b49-b18b-8cf6665ee0a9" (UID: "99646111-522d-4b49-b18b-8cf6665ee0a9"). InnerVolumeSpecName "kube-api-access-4kp6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.770741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.770709    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kp6p\" (UniqueName: \"kubernetes.io/projected/99646111-522d-4b49-b18b-8cf6665ee0a9-kube-api-access-4kp6p\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.770741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:28.770737    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99646111-522d-4b49-b18b-8cf6665ee0a9-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.325744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:29.325654    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:29.388258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:29.388230    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"
Oct 30 22:28:29.388607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:29.388272    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv" event={"ID":"99646111-522d-4b49-b18b-8cf6665ee0a9","Type":"ContainerDied","Data":"6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47"}
Oct 30 22:28:29.388607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:29.388321    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce06922f538167cbd811d9079abf7ad5e25b7bfcc81d847244e652c60ac5a47"
Oct 30 22:28:32.317102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:32.317065    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:36.168191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:36.168155    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:28:36.566879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:36.566798    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:36.781942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:36.781903    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:38.386649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:38.386617    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:39.853158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853123    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"]
Oct 30 22:28:39.853807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853748    4410 cpu_manager.go:414] "containerMap: removing container" podUID="99646111-522d-4b49-b18b-8cf6665ee0a9" containerName="revoke-keys"
Oct 30 22:28:39.853807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853787    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99646111-522d-4b49-b18b-8cf6665ee0a9" containerName="revoke-keys"
Oct 30 22:28:39.853980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853834    4410 cpu_manager.go:414] "containerMap: removing container" podUID="745987a6-cba7-49c4-95b6-bac1775035c0" containerName="revoke-keys"
Oct 30 22:28:39.853980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853844    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745987a6-cba7-49c4-95b6-bac1775035c0" containerName="revoke-keys"
Oct 30 22:28:39.853980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853961    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="745987a6-cba7-49c4-95b6-bac1775035c0" containerName="revoke-keys"
Oct 30 22:28:39.853980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.853977    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="99646111-522d-4b49-b18b-8cf6665ee0a9" containerName="revoke-keys"
Oct 30 22:28:39.858440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.858418    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.886599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.886567    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4mz\" (UniqueName: \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.886722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.886640    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.886848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.886826    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.914946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.914917    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"]
Oct 30 22:28:39.960502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.960475    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:39.960634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.960622    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:28:39.960676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.960635    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:39.960676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.960627    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:39.987671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.987642    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4mz\" (UniqueName: \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.987831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.987710    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.987904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.987873    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.988385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.988368    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:39.990162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:39.990145    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:40.055301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:40.055279    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4mz\" (UniqueName: \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\") pod \"maas-api-revoke-sub-1fcd97ba-px66m\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:40.171457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:40.171434    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:40.545382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:40.545358    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"]
Oct 30 22:28:40.546881 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:28:40.546853    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc291032_1a5e_4a2d_a121_114d14bc8bbb.slice/crio-713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51 WatchSource:0}: Error finding container 713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51: Status 404 returned error can't find the container with id 713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51
Oct 30 22:28:40.784383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:40.784353    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.165530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.165494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.168650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.168603    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" podUID="8ccf9561-271c-449f-83a2-16d1ad811ad6" containerName="payload-processing" containerID="cri-o://05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6" gracePeriod=25
Oct 30 22:28:41.252473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.252445    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"]
Oct 30 22:28:41.302007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.301974    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5b1e39-fm9fv"]
Oct 30 22:28:41.425012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.424958    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:28:41.459099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.459072    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m" event={"ID":"cc291032-1a5e-4a2d-a121-114d14bc8bbb","Type":"ContainerStarted","Data":"05cf53b85f55ac38423335aee60ce9fee3dd7cc65a8ee7cda75586944dd67173"}
Oct 30 22:28:41.459197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.459128    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m" event={"ID":"cc291032-1a5e-4a2d-a121-114d14bc8bbb","Type":"ContainerStarted","Data":"713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51"}
Oct 30 22:28:41.460351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.460328    4410 generic.go:356] "Generic (PLEG): container finished" podID="8ccf9561-271c-449f-83a2-16d1ad811ad6" containerID="05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6" exitCode=0
Oct 30 22:28:41.460451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.460366    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"
Oct 30 22:28:41.460451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.460409    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" event={"ID":"8ccf9561-271c-449f-83a2-16d1ad811ad6","Type":"ContainerDied","Data":"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"}
Oct 30 22:28:41.460536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.460450    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s" event={"ID":"8ccf9561-271c-449f-83a2-16d1ad811ad6","Type":"ContainerDied","Data":"db6b0253383d1531a5be623ebc5e735548a139f3272b0cfa5de013e1e6289790"}
Oct 30 22:28:41.460536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.460472    4410 scope.go:122] "RemoveContainer" containerID="05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"
Oct 30 22:28:41.466518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.466498    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.505063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.505033    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\" (UniqueName: \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\") pod \"8ccf9561-271c-449f-83a2-16d1ad811ad6\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") "
Oct 30 22:28:41.505226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.505212    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\") pod \"8ccf9561-271c-449f-83a2-16d1ad811ad6\" (UID: \"8ccf9561-271c-449f-83a2-16d1ad811ad6\") "
Oct 30 22:28:41.505568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.505545    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume" pod "8ccf9561-271c-449f-83a2-16d1ad811ad6" (UID: "8ccf9561-271c-449f-83a2-16d1ad811ad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:41.507111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.507088    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f" pod "8ccf9561-271c-449f-83a2-16d1ad811ad6" (UID: "8ccf9561-271c-449f-83a2-16d1ad811ad6"). InnerVolumeSpecName "kube-api-access-j6f5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:41.528396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.528374    4410 scope.go:122] "RemoveContainer" containerID="05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"
Oct 30 22:28:41.528725 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:28:41.528698    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6\": container with ID starting with 05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6 not found: ID does not exist" containerID="05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"
Oct 30 22:28:41.528795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.528734    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6"} err="failed to get container status \"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6\": rpc error: code = NotFound desc = could not find container \"05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6\": container with ID starting with 05f3b6d52fc50141bc0d72a89861781ec78c99c4ca9a9ec505ea0b2f569e39d6 not found: ID does not exist"
Oct 30 22:28:41.606676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.606645    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccf9561-271c-449f-83a2-16d1ad811ad6-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.606676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.606671    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6f5f\" (UniqueName: \"kubernetes.io/projected/8ccf9561-271c-449f-83a2-16d1ad811ad6-kube-api-access-j6f5f\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.807075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.807040    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.882717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.882685    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:28:41.890388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.890363    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5b1e39-6557cdbb99-zps2s"]
Oct 30 22:28:41.998875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:41.998847    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:42.467954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.467921    4410 generic.go:356] "Generic (PLEG): container finished" podID="cc291032-1a5e-4a2d-a121-114d14bc8bbb" containerID="05cf53b85f55ac38423335aee60ce9fee3dd7cc65a8ee7cda75586944dd67173" exitCode=0
Oct 30 22:28:42.468363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.468009    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m" event={"ID":"cc291032-1a5e-4a2d-a121-114d14bc8bbb","Type":"ContainerDied","Data":"05cf53b85f55ac38423335aee60ce9fee3dd7cc65a8ee7cda75586944dd67173"}
Oct 30 22:28:42.613075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.613045    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:42.718658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.718573    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\" (UniqueName: \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\") pod \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") "
Oct 30 22:28:42.718838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.718723    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\") pod \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") "
Oct 30 22:28:42.718838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.718758    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\") pod \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\" (UID: \"cc291032-1a5e-4a2d-a121-114d14bc8bbb\") "
Oct 30 22:28:42.719111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.719084    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca" pod "cc291032-1a5e-4a2d-a121-114d14bc8bbb" (UID: "cc291032-1a5e-4a2d-a121-114d14bc8bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:42.719208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.719168    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.720789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.720751    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz" pod "cc291032-1a5e-4a2d-a121-114d14bc8bbb" (UID: "cc291032-1a5e-4a2d-a121-114d14bc8bbb"). InnerVolumeSpecName "kube-api-access-qk4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:42.720880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.720837    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca" pod "cc291032-1a5e-4a2d-a121-114d14bc8bbb" (UID: "cc291032-1a5e-4a2d-a121-114d14bc8bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:42.820250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.820210    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4mz\" (UniqueName: \"kubernetes.io/projected/cc291032-1a5e-4a2d-a121-114d14bc8bbb-kube-api-access-qk4mz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.820250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:42.820248    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc291032-1a5e-4a2d-a121-114d14bc8bbb-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:43.020347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.020263    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccf9561-271c-449f-83a2-16d1ad811ad6" path="/var/lib/kubelet/pods/8ccf9561-271c-449f-83a2-16d1ad811ad6/volumes"
Oct 30 22:28:43.020856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.020840    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99646111-522d-4b49-b18b-8cf6665ee0a9" path="/var/lib/kubelet/pods/99646111-522d-4b49-b18b-8cf6665ee0a9/volumes"
Oct 30 22:28:43.312578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.312489    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:43.477834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.477791    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m" event={"ID":"cc291032-1a5e-4a2d-a121-114d14bc8bbb","Type":"ContainerDied","Data":"713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51"}
Oct 30 22:28:43.477834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.477829    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"
Oct 30 22:28:43.478285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.477849    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="713ada120145e186838a7f29d1080e87048aec0f1c4a9168609798f876eeae51"
Oct 30 22:28:43.520736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:43.520701    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.024433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:48.024401    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.190394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:48.190359    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.385226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:48.385143    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.547656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:48.547620    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:50.523755 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:28:50.523718    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:28:53.446958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:53.446928    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:58.399974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:28:58.399936    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:02.672089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672056    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"]
Oct 30 22:29:02.672515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672502    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cc291032-1a5e-4a2d-a121-114d14bc8bbb" containerName="revoke-keys"
Oct 30 22:29:02.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672516    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc291032-1a5e-4a2d-a121-114d14bc8bbb" containerName="revoke-keys"
Oct 30 22:29:02.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672532    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8ccf9561-271c-449f-83a2-16d1ad811ad6" containerName="payload-processing"
Oct 30 22:29:02.672560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672536    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccf9561-271c-449f-83a2-16d1ad811ad6" containerName="payload-processing"
Oct 30 22:29:02.672658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672607    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc291032-1a5e-4a2d-a121-114d14bc8bbb" containerName="revoke-keys"
Oct 30 22:29:02.672658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.672618    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccf9561-271c-449f-83a2-16d1ad811ad6" containerName="payload-processing"
Oct 30 22:29:02.675911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.675893    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.732669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.732638    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"]
Oct 30 22:29:02.763814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.763788    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.763973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.763946    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.764017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.763995    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.778755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.778732    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:02.778875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.778765    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:02.778875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.778765    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:02.778964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.778871    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:02.864937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.864912    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.865069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.864977    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.865069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.864998    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.865673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.865655    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.867241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.867221    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.921997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.921972    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\") pod \"maas-api-revoke-sub-f9322e0e-2kt6q\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:02.989171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:02.989115    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:03.345729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:03.345629    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.367681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:03.367647    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"]
Oct 30 22:29:03.375655 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:03.371651    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc4096a_1a52_435a_b875_e39bbc39f0ed.slice/crio-0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36 WatchSource:0}: Error finding container 0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36: Status 404 returned error can't find the container with id 0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36
Oct 30 22:29:03.632489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:03.632421    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q" event={"ID":"7bc4096a-1a52-435a-b875-e39bbc39f0ed","Type":"ContainerStarted","Data":"0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36"}
Oct 30 22:29:04.640462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:04.640425    4410 generic.go:356] "Generic (PLEG): container finished" podID="7bc4096a-1a52-435a-b875-e39bbc39f0ed" containerID="5955cd17acfc2e9c3cf5916c7d4d598bb4f03c458d41912b38afe375d332b180" exitCode=0
Oct 30 22:29:04.640880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:04.640514    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q" event={"ID":"7bc4096a-1a52-435a-b875-e39bbc39f0ed","Type":"ContainerDied","Data":"5955cd17acfc2e9c3cf5916c7d4d598bb4f03c458d41912b38afe375d332b180"}
Oct 30 22:29:05.790263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.790242    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:05.899687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.899624    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\") pod \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") "
Oct 30 22:29:05.899687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.899673    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\") pod \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") "
Oct 30 22:29:05.899917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.899788    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\") pod \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\" (UID: \"7bc4096a-1a52-435a-b875-e39bbc39f0ed\") "
Oct 30 22:29:05.900120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.900092    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca" pod "7bc4096a-1a52-435a-b875-e39bbc39f0ed" (UID: "7bc4096a-1a52-435a-b875-e39bbc39f0ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:05.901586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.901569    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4" pod "7bc4096a-1a52-435a-b875-e39bbc39f0ed" (UID: "7bc4096a-1a52-435a-b875-e39bbc39f0ed"). InnerVolumeSpecName "kube-api-access-w4bj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:05.901683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:05.901664    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca" pod "7bc4096a-1a52-435a-b875-e39bbc39f0ed" (UID: "7bc4096a-1a52-435a-b875-e39bbc39f0ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:06.001299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.001256    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/7bc4096a-1a52-435a-b875-e39bbc39f0ed-kube-api-access-w4bj4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.001299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.001289    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.001299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.001299    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4096a-1a52-435a-b875-e39bbc39f0ed-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.653098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.653066    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q" event={"ID":"7bc4096a-1a52-435a-b875-e39bbc39f0ed","Type":"ContainerDied","Data":"0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36"}
Oct 30 22:29:06.653098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.653094    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"
Oct 30 22:29:06.653340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:06.653112    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a088ef8b62b0f7ba09c9f268bcdad0ef4d4bcca3ad639b9ae079ba016bdac36"
Oct 30 22:29:08.171099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:08.171065    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:12.970507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:12.970471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.349159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.349128    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"]
Oct 30 22:29:15.349579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.349546    4410 cpu_manager.go:414] "containerMap: removing container" podUID="7bc4096a-1a52-435a-b875-e39bbc39f0ed" containerName="revoke-keys"
Oct 30 22:29:15.349579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.349563    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc4096a-1a52-435a-b875-e39bbc39f0ed" containerName="revoke-keys"
Oct 30 22:29:15.349727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.349715    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc4096a-1a52-435a-b875-e39bbc39f0ed" containerName="revoke-keys"
Oct 30 22:29:15.353036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.353019    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.410257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.410224    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"]
Oct 30 22:29:15.455615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.455584    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:15.455883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.455866    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:15.455952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.455908    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:15.455952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.455945    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:15.497707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.497675    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.497876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.497842    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.497935 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.497915    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hj8x\" (UniqueName: \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.598480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.598449    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.598628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.598533    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.598628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.598569    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hj8x\" (UniqueName: \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.599266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.599205    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.600909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.600888    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.655305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.655276    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hj8x\" (UniqueName: \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\") pod \"maas-api-revoke-sub-d3ecc667-58tn4\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:15.665866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:15.665842    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:16.040993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:16.040970    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"]
Oct 30 22:29:16.042376 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:16.042349    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2043cadd_50ab_4024_9d6d_5a81deb32668.slice/crio-9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6 WatchSource:0}: Error finding container 9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6: Status 404 returned error can't find the container with id 9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6
Oct 30 22:29:16.715942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:16.715906    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4" event={"ID":"2043cadd-50ab-4024-9d6d-5a81deb32668","Type":"ContainerStarted","Data":"1b10d30de0100a4e19a930ff77f672a212040427ab6bf87a7ef013404c7be92d"}
Oct 30 22:29:16.716330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:16.715987    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4" event={"ID":"2043cadd-50ab-4024-9d6d-5a81deb32668","Type":"ContainerStarted","Data":"9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6"}
Oct 30 22:29:16.832145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:16.832070    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4" podStartSLOduration=1.8320565260000001 podStartE2EDuration="1.832056526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:16.829847916 +0000 UTC m=+2809.918585718" watchObservedRunningTime="2026-10-30 22:29:16.832056526 +0000 UTC m=+2809.920794328"
Oct 30 22:29:17.723179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:17.723140    4410 generic.go:356] "Generic (PLEG): container finished" podID="2043cadd-50ab-4024-9d6d-5a81deb32668" containerID="1b10d30de0100a4e19a930ff77f672a212040427ab6bf87a7ef013404c7be92d" exitCode=0
Oct 30 22:29:17.723620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:17.723225    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4" event={"ID":"2043cadd-50ab-4024-9d6d-5a81deb32668","Type":"ContainerDied","Data":"1b10d30de0100a4e19a930ff77f672a212040427ab6bf87a7ef013404c7be92d"}
Oct 30 22:29:17.869163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:17.869127    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:18.836984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:18.836951    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:18.892967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:18.892942    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:19.036043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.035960    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\") pod \"2043cadd-50ab-4024-9d6d-5a81deb32668\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") "
Oct 30 22:29:19.036214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.036099    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\") pod \"2043cadd-50ab-4024-9d6d-5a81deb32668\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") "
Oct 30 22:29:19.036286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.036260    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\" (UniqueName: \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\") pod \"2043cadd-50ab-4024-9d6d-5a81deb32668\" (UID: \"2043cadd-50ab-4024-9d6d-5a81deb32668\") "
Oct 30 22:29:19.036286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.036272    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca" pod "2043cadd-50ab-4024-9d6d-5a81deb32668" (UID: "2043cadd-50ab-4024-9d6d-5a81deb32668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:19.036730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.036707    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:19.038277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.038246    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca" pod "2043cadd-50ab-4024-9d6d-5a81deb32668" (UID: "2043cadd-50ab-4024-9d6d-5a81deb32668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:19.038554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.038531    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x" pod "2043cadd-50ab-4024-9d6d-5a81deb32668" (UID: "2043cadd-50ab-4024-9d6d-5a81deb32668"). InnerVolumeSpecName "kube-api-access-9hj8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:19.137417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.137381    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043cadd-50ab-4024-9d6d-5a81deb32668-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:19.137417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.137409    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hj8x\" (UniqueName: \"kubernetes.io/projected/2043cadd-50ab-4024-9d6d-5a81deb32668-kube-api-access-9hj8x\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:19.737208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.737168    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4" event={"ID":"2043cadd-50ab-4024-9d6d-5a81deb32668","Type":"ContainerDied","Data":"9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6"}
Oct 30 22:29:19.737380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.737250    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a28c824360e247dbf2b61e344d6791458bed2d87e01158779908d1f9183f3c6"
Oct 30 22:29:19.737380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.737191    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"
Oct 30 22:29:19.988820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:19.988718    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.567030 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:29:20.566984    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:29:20.623228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:20.623194    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.815035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:20.814993    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:22.907957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:22.907927    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.340559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:24.340522    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.528814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:24.528783    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.693924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:24.693893    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.757911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.757878    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"]
Oct 30 22:29:25.758322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.758309    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2043cadd-50ab-4024-9d6d-5a81deb32668" containerName="revoke-keys"
Oct 30 22:29:25.758373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.758323    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2043cadd-50ab-4024-9d6d-5a81deb32668" containerName="revoke-keys"
Oct 30 22:29:25.758415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.758401    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2043cadd-50ab-4024-9d6d-5a81deb32668" containerName="revoke-keys"
Oct 30 22:29:25.762165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.762140    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.814660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.814638    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:25.814839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.814822    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:25.817481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.817459    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5jp\" (UniqueName: \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.817583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.817568    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.817625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.817606    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.864009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.863975    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:25.864172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.864070    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:25.867151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.867129    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"]
Oct 30 22:29:25.918868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.918842    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5jp\" (UniqueName: \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.918999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.918918    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.918999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.918948    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.919547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.919523    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.921278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.921255    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:25.975167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:25.975144    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5jp\" (UniqueName: \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\") pod \"maas-api-revoke-sub-781d01ed-h5jdx\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:26.075838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.075753    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:26.448922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.448894    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"]
Oct 30 22:29:26.450681 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:26.450652    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4a5589_fef2_43b2_b087_354036a65040.slice/crio-eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0 WatchSource:0}: Error finding container eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0: Status 404 returned error can't find the container with id eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0
Oct 30 22:29:26.597556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.597527    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:26.780908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.780882    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx" event={"ID":"1b4a5589-fef2-43b2-b087-354036a65040","Type":"ContainerStarted","Data":"352902ee61f1c34da1758277df347886d6e8a55a06778d972dc1a4ed3c3378e2"}
Oct 30 22:29:26.781225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.780932    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx" event={"ID":"1b4a5589-fef2-43b2-b087-354036a65040","Type":"ContainerStarted","Data":"eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0"}
Oct 30 22:29:26.898010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:26.897946    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx" podStartSLOduration=1.897924741 podStartE2EDuration="1.897924741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:26.894521873 +0000 UTC m=+2819.983259675" watchObservedRunningTime="2026-10-30 22:29:26.897924741 +0000 UTC m=+2819.986662545"
Oct 30 22:29:27.692324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:27.692280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.797920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:27.797873    4410 generic.go:356] "Generic (PLEG): container finished" podID="1b4a5589-fef2-43b2-b087-354036a65040" containerID="352902ee61f1c34da1758277df347886d6e8a55a06778d972dc1a4ed3c3378e2" exitCode=0
Oct 30 22:29:27.798406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:27.797953    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx" event={"ID":"1b4a5589-fef2-43b2-b087-354036a65040","Type":"ContainerDied","Data":"352902ee61f1c34da1758277df347886d6e8a55a06778d972dc1a4ed3c3378e2"}
Oct 30 22:29:27.884414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:27.884328    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:28.324796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.307566    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:29:28.349803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.326969    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:28.350138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.337021    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:29:28.350317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.337115    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.419062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.416082    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3bbd9b\""
Oct 30 22:29:28.419062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.416823    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3bbd9b-dockercfg-wgp4k\""
Oct 30 22:29:28.439521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.439490    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.439860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.439833    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494cg\" (UniqueName: \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.541566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.541531    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494cg\" (UniqueName: \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.541711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.541653    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.542377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.542353    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.615853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.609381    4410 scope.go:122] "RemoveContainer" containerID="2aa714a137bad41a4d780cfed340625f308042087c61e7564ad0619f148d9268"
Oct 30 22:29:28.620316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.620251    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-494cg\" (UniqueName: \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\") pod \"payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:28.633622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.633596    4410 scope.go:122] "RemoveContainer" containerID="1feacd6120eb1a0eb00011a69b97b90f2f00eb5a6328d2a221749c38f29c432e"
Oct 30 22:29:28.642481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.642464    4410 scope.go:122] "RemoveContainer" containerID="96c2e5cc834ee2dc4cbd6d1c87a8ee9583b6a90c2d94b3c9a59a119415a5891d"
Oct 30 22:29:28.721412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:28.721149    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:29.108477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.108452    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:29.163450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.163374    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\" (UniqueName: \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\") pod \"1b4a5589-fef2-43b2-b087-354036a65040\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") "
Oct 30 22:29:29.163620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.163599    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\") pod \"1b4a5589-fef2-43b2-b087-354036a65040\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") "
Oct 30 22:29:29.163689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.163642    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\") pod \"1b4a5589-fef2-43b2-b087-354036a65040\" (UID: \"1b4a5589-fef2-43b2-b087-354036a65040\") "
Oct 30 22:29:29.164069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.164044    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca" pod "1b4a5589-fef2-43b2-b087-354036a65040" (UID: "1b4a5589-fef2-43b2-b087-354036a65040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:29.165553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.165528    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp" pod "1b4a5589-fef2-43b2-b087-354036a65040" (UID: "1b4a5589-fef2-43b2-b087-354036a65040"). InnerVolumeSpecName "kube-api-access-mg5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:29.165639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.165621    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca" pod "1b4a5589-fef2-43b2-b087-354036a65040" (UID: "1b4a5589-fef2-43b2-b087-354036a65040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:29.215891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.215858    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:29:29.216721 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:29.216692    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b78d935_14be_43ba_91fd_6f9d8eec7248.slice/crio-165c5bad727bb8e1ea2e9982bd80f4b51b2040b53294e6064850325af6f7e698 WatchSource:0}: Error finding container 165c5bad727bb8e1ea2e9982bd80f4b51b2040b53294e6064850325af6f7e698: Status 404 returned error can't find the container with id 165c5bad727bb8e1ea2e9982bd80f4b51b2040b53294e6064850325af6f7e698
Oct 30 22:29:29.264652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.264626    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.264790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.264655    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4a5589-fef2-43b2-b087-354036a65040-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.264790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.264671    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5jp\" (UniqueName: \"kubernetes.io/projected/1b4a5589-fef2-43b2-b087-354036a65040-kube-api-access-mg5jp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.659090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.659059    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.845897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.845867    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"
Oct 30 22:29:29.846195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.845866    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx" event={"ID":"1b4a5589-fef2-43b2-b087-354036a65040","Type":"ContainerDied","Data":"eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0"}
Oct 30 22:29:29.846349 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.846263    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb83907a885de7b75eba470ef445c5736f9a82e8f982c5f9358d74cf51ffdcf0"
Oct 30 22:29:29.847908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.847877    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" event={"ID":"8b78d935-14be-43ba-91fd-6f9d8eec7248","Type":"ContainerStarted","Data":"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"}
Oct 30 22:29:29.848050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.847930    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" event={"ID":"8b78d935-14be-43ba-91fd-6f9d8eec7248","Type":"ContainerStarted","Data":"165c5bad727bb8e1ea2e9982bd80f4b51b2040b53294e6064850325af6f7e698"}
Oct 30 22:29:29.848050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.847994    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:29.962417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:29.962359    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" podStartSLOduration=1.962346226 podStartE2EDuration="1.962346226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:29.961400379 +0000 UTC m=+2823.050138193" watchObservedRunningTime="2026-10-30 22:29:29.962346226 +0000 UTC m=+2823.051084029"
Oct 30 22:29:31.065767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.065730    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"]
Oct 30 22:29:31.066279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.066227    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1b4a5589-fef2-43b2-b087-354036a65040" containerName="revoke-keys"
Oct 30 22:29:31.066279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.066241    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4a5589-fef2-43b2-b087-354036a65040" containerName="revoke-keys"
Oct 30 22:29:31.066382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.066325    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4a5589-fef2-43b2-b087-354036a65040" containerName="revoke-keys"
Oct 30 22:29:31.069685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.069665    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.123745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.123717    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:31.172969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.172939    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:31.173194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.172947    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:31.173313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.173040    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:31.176181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.176146    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"]
Oct 30 22:29:31.184263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.184239    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.184381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.184357    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.184456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.184438    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mfcv\" (UniqueName: \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.286111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.286075    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.286282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.286159    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfcv\" (UniqueName: \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.286333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.286298    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.286922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.286898    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.288399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.288377    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.343330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.343273    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mfcv\" (UniqueName: \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\") pod \"maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.383265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.383241    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:31.762493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.760656    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"]
Oct 30 22:29:31.764171 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:31.764141    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6966b15_4705_41ec_b43d_294724484249.slice/crio-e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a WatchSource:0}: Error finding container e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a: Status 404 returned error can't find the container with id e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a
Oct 30 22:29:31.860831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:31.860797    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj" event={"ID":"e6966b15-4705-41ec-b43d-294724484249","Type":"ContainerStarted","Data":"e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a"}
Oct 30 22:29:32.869244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:32.869154    4410 generic.go:356] "Generic (PLEG): container finished" podID="e6966b15-4705-41ec-b43d-294724484249" containerID="efeb624bdea78ed65fcea12c33fd638d7d077b9b646482b8476bcec3b69592df" exitCode=0
Oct 30 22:29:32.869244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:32.869201    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj" event={"ID":"e6966b15-4705-41ec-b43d-294724484249","Type":"ContainerDied","Data":"efeb624bdea78ed65fcea12c33fd638d7d077b9b646482b8476bcec3b69592df"}
Oct 30 22:29:32.922433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:32.922399    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:33.632896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:33.632863    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.018654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.018630    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:34.119600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.119576    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\") pod \"e6966b15-4705-41ec-b43d-294724484249\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") "
Oct 30 22:29:34.119752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.119654    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\" (UniqueName: \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\") pod \"e6966b15-4705-41ec-b43d-294724484249\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") "
Oct 30 22:29:34.119752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.119692    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\") pod \"e6966b15-4705-41ec-b43d-294724484249\" (UID: \"e6966b15-4705-41ec-b43d-294724484249\") "
Oct 30 22:29:34.120109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.120075    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca" pod "e6966b15-4705-41ec-b43d-294724484249" (UID: "e6966b15-4705-41ec-b43d-294724484249"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:34.120213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.120164    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6966b15-4705-41ec-b43d-294724484249-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:34.121680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.121653    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca" pod "e6966b15-4705-41ec-b43d-294724484249" (UID: "e6966b15-4705-41ec-b43d-294724484249"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:34.121680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.121665    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv" pod "e6966b15-4705-41ec-b43d-294724484249" (UID: "e6966b15-4705-41ec-b43d-294724484249"). InnerVolumeSpecName "kube-api-access-4mfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:34.221710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.221637    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6966b15-4705-41ec-b43d-294724484249-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:34.221710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.221662    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mfcv\" (UniqueName: \"kubernetes.io/projected/e6966b15-4705-41ec-b43d-294724484249-kube-api-access-4mfcv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:34.881615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.881578    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj" event={"ID":"e6966b15-4705-41ec-b43d-294724484249","Type":"ContainerDied","Data":"e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a"}
Oct 30 22:29:34.881800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.881665    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9a99041674f9f90f5608b948fb28cd692fcd54d1ee80fb4a2c0a7a683b03c7a"
Oct 30 22:29:34.881800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:34.881591    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"
Oct 30 22:29:38.753679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:38.753638    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:40.857119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:40.857042    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:29:41.414830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.414801    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"]
Oct 30 22:29:41.415245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.415233    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e6966b15-4705-41ec-b43d-294724484249" containerName="revoke-keys"
Oct 30 22:29:41.415290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.415246    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6966b15-4705-41ec-b43d-294724484249" containerName="revoke-keys"
Oct 30 22:29:41.415333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.415329    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6966b15-4705-41ec-b43d-294724484249" containerName="revoke-keys"
Oct 30 22:29:41.418665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.418650    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.472350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.472300    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:41.497674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.497646    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.497864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.497683    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.497864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.497719    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b54\" (UniqueName: \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.521071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.521046    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:41.521189 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.521167    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:41.521227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.521167    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:41.524387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.524366    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"]
Oct 30 22:29:41.599125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.599099    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.599299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.599137    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.599299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.599165    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b54\" (UniqueName: \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.599829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.599806    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.601367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.601346    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.655948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.655925    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b54\" (UniqueName: \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\") pod \"maas-api-revoke-sub-95ef73e4-pqgs2\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:41.731874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:41.731815    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:42.114575 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:42.114538    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2493766f_43ed_4f88_a24c_2e083b33b45b.slice/crio-076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2 WatchSource:0}: Error finding container 076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2: Status 404 returned error can't find the container with id 076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2
Oct 30 22:29:42.114959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:42.114642    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"]
Oct 30 22:29:42.619102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:42.619071    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:29:42.726925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:42.726857    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:29:42.932514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:42.932482    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2" event={"ID":"2493766f-43ed-4f88-a24c-2e083b33b45b","Type":"ContainerStarted","Data":"1e6e6e722671b5c5653ac22cdde28884842c947a86bab13add086ac3d0bb77b6"}
Oct 30 22:29:42.932696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:42.932543    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2" event={"ID":"2493766f-43ed-4f88-a24c-2e083b33b45b","Type":"ContainerStarted","Data":"076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2"}
Oct 30 22:29:43.000596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.000513    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.046370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.046311    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2" podStartSLOduration=2.046291978 podStartE2EDuration="2.046291978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:43.046018071 +0000 UTC m=+2836.134755880" watchObservedRunningTime="2026-10-30 22:29:43.046291978 +0000 UTC m=+2836.135029787"
Oct 30 22:29:43.198638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.198596    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.669236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.669201    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.940506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.940426    4410 generic.go:356] "Generic (PLEG): container finished" podID="2493766f-43ed-4f88-a24c-2e083b33b45b" containerID="1e6e6e722671b5c5653ac22cdde28884842c947a86bab13add086ac3d0bb77b6" exitCode=0
Oct 30 22:29:43.940732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:43.940490    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2" event={"ID":"2493766f-43ed-4f88-a24c-2e083b33b45b","Type":"ContainerDied","Data":"1e6e6e722671b5c5653ac22cdde28884842c947a86bab13add086ac3d0bb77b6"}
Oct 30 22:29:45.096936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.096912    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:45.243034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.242953    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\") pod \"2493766f-43ed-4f88-a24c-2e083b33b45b\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") "
Oct 30 22:29:45.243034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.243009    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\" (UniqueName: \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\") pod \"2493766f-43ed-4f88-a24c-2e083b33b45b\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") "
Oct 30 22:29:45.243270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.243077    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\") pod \"2493766f-43ed-4f88-a24c-2e083b33b45b\" (UID: \"2493766f-43ed-4f88-a24c-2e083b33b45b\") "
Oct 30 22:29:45.243471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.243440    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca" pod "2493766f-43ed-4f88-a24c-2e083b33b45b" (UID: "2493766f-43ed-4f88-a24c-2e083b33b45b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:45.245145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.245120    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca" pod "2493766f-43ed-4f88-a24c-2e083b33b45b" (UID: "2493766f-43ed-4f88-a24c-2e083b33b45b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:45.245256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.245237    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54" pod "2493766f-43ed-4f88-a24c-2e083b33b45b" (UID: "2493766f-43ed-4f88-a24c-2e083b33b45b"). InnerVolumeSpecName "kube-api-access-r7b54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:45.345005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.344980    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.345005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.345002    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7b54\" (UniqueName: \"kubernetes.io/projected/2493766f-43ed-4f88-a24c-2e083b33b45b-kube-api-access-r7b54\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.345148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.345012    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2493766f-43ed-4f88-a24c-2e083b33b45b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.953165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.953137    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"
Oct 30 22:29:45.953341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.953165    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2" event={"ID":"2493766f-43ed-4f88-a24c-2e083b33b45b","Type":"ContainerDied","Data":"076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2"}
Oct 30 22:29:45.953341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:45.953216    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076056c0e72519d429e1b64e18a0aa0ad25da4a48073e3eca895e4211258f2d2"
Oct 30 22:29:46.671793 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.671754    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"]
Oct 30 22:29:46.672211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.672197    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2493766f-43ed-4f88-a24c-2e083b33b45b" containerName="revoke-keys"
Oct 30 22:29:46.672256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.672211    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2493766f-43ed-4f88-a24c-2e083b33b45b" containerName="revoke-keys"
Oct 30 22:29:46.672311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.672302    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2493766f-43ed-4f88-a24c-2e083b33b45b" containerName="revoke-keys"
Oct 30 22:29:46.675488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.675469    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.733979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.733951    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"]
Oct 30 22:29:46.778800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.778745    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:46.778800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.778753    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:46.779011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.778830    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:46.779011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.778868    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:46.860440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.860400    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.860606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.860571    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.860729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.860714    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwmd\" (UniqueName: \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.962027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.961944    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.962172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.962051    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwmd\" (UniqueName: \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.962172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.962111    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.962643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.962621    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:46.964195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:46.964177    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:47.018627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.018597    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwmd\" (UniqueName: \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\") pod \"maas-api-revoke-sub-f66928bd-tm697\" (UID: \"38641418-2105-4296-b549-05083e61662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:47.289008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.288926    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:47.619426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.619346    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing" containerID="cri-o://4551b481fa51dc88ca66b0e02e32fe046a56bbfb948ad3964ea2e0c3a7e9afd2" gracePeriod=25
Oct 30 22:29:47.667452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.667424    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"]
Oct 30 22:29:47.668666 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:47.668639    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38641418_2105_4296_b549_05083e61662d.slice/crio-4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea WatchSource:0}: Error finding container 4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea: Status 404 returned error can't find the container with id 4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea
Oct 30 22:29:47.727384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.727350    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing" containerID="cri-o://86ce50f47d9588534b0133800e8ecf46f08b8cad70f7c6ef6f1247e6caafac5c" gracePeriod=25
Oct 30 22:29:47.966000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.965966    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697" event={"ID":"38641418-2105-4296-b549-05083e61662d","Type":"ContainerStarted","Data":"03b7c0dff631b9c682b090173b9eab35d2f0bbdcccfd45cb8647f6108817a66c"}
Oct 30 22:29:47.966111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:47.966031    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697" event={"ID":"38641418-2105-4296-b549-05083e61662d","Type":"ContainerStarted","Data":"4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea"}
Oct 30 22:29:48.079607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:48.079556    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697" podStartSLOduration=2.07954384 podStartE2EDuration="2.07954384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:48.07868154 +0000 UTC m=+2841.167419345" watchObservedRunningTime="2026-10-30 22:29:48.07954384 +0000 UTC m=+2841.168281642"
Oct 30 22:29:48.762308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:48.762233    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.973272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:48.973233    4410 generic.go:356] "Generic (PLEG): container finished" podID="38641418-2105-4296-b549-05083e61662d" containerID="03b7c0dff631b9c682b090173b9eab35d2f0bbdcccfd45cb8647f6108817a66c" exitCode=0
Oct 30 22:29:48.973450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:48.973320    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697" event={"ID":"38641418-2105-4296-b549-05083e61662d","Type":"ContainerDied","Data":"03b7c0dff631b9c682b090173b9eab35d2f0bbdcccfd45cb8647f6108817a66c"}
Oct 30 22:29:50.135740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.135715    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:50.197162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.197132    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\" (UniqueName: \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\") pod \"38641418-2105-4296-b549-05083e61662d\" (UID: \"38641418-2105-4296-b549-05083e61662d\") "
Oct 30 22:29:50.197308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.197172    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\") pod \"38641418-2105-4296-b549-05083e61662d\" (UID: \"38641418-2105-4296-b549-05083e61662d\") "
Oct 30 22:29:50.197308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.197210    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\") pod \"38641418-2105-4296-b549-05083e61662d\" (UID: \"38641418-2105-4296-b549-05083e61662d\") "
Oct 30 22:29:50.197584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.197560    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca" pod "38641418-2105-4296-b549-05083e61662d" (UID: "38641418-2105-4296-b549-05083e61662d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:50.197658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.197644    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38641418-2105-4296-b549-05083e61662d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:50.199278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.199249    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca" pod "38641418-2105-4296-b549-05083e61662d" (UID: "38641418-2105-4296-b549-05083e61662d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:50.199397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.199292    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd" pod "38641418-2105-4296-b549-05083e61662d" (UID: "38641418-2105-4296-b549-05083e61662d"). InnerVolumeSpecName "kube-api-access-8pwmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:50.298626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.298538    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwmd\" (UniqueName: \"kubernetes.io/projected/38641418-2105-4296-b549-05083e61662d-kube-api-access-8pwmd\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:50.298626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.298571    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38641418-2105-4296-b549-05083e61662d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:50.522237 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:29:50.522202    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:29:50.987305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.987278    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"
Oct 30 22:29:50.987487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.987314    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697" event={"ID":"38641418-2105-4296-b549-05083e61662d","Type":"ContainerDied","Data":"4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea"}
Oct 30 22:29:50.987487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:50.987362    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4447bbbddca775c60820d03b827a0ff8e1372e8c4fdb2211d8e224109309cbea"
Oct 30 22:29:53.070243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.070213    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"]
Oct 30 22:29:53.070668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.070655    4410 cpu_manager.go:414] "containerMap: removing container" podUID="38641418-2105-4296-b549-05083e61662d" containerName="revoke-keys"
Oct 30 22:29:53.070709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.070669    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38641418-2105-4296-b549-05083e61662d" containerName="revoke-keys"
Oct 30 22:29:53.070754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.070744    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="38641418-2105-4296-b549-05083e61662d" containerName="revoke-keys"
Oct 30 22:29:53.074432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.074413    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.130893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.130864    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"]
Oct 30 22:29:53.175601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.175572    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:53.175737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.175615    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:53.175737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.175624    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:29:53.175737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.175663    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:53.197323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.197297    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.197450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.197346    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.197450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.197416    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl8b\" (UniqueName: \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.298976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.298950    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.299106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.299002    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.299106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.299025    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl8b\" (UniqueName: \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.299612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.299593    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.301359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.301339    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.355486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.355427    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl8b\" (UniqueName: \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\") pod \"maas-api-revoke-sub-69330c9c-vzq75\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.387308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.387276    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:53.767533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.767502    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"]
Oct 30 22:29:53.769503 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:53.769472    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593b2375_c50d_4de0_aca8_57a8df512166.slice/crio-f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208 WatchSource:0}: Error finding container f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208: Status 404 returned error can't find the container with id f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208
Oct 30 22:29:53.855585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:53.855552    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:54.030379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:54.030308    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75" event={"ID":"593b2375-c50d-4de0-aca8-57a8df512166","Type":"ContainerStarted","Data":"f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208"}
Oct 30 22:29:54.786736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:54.786701    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:54.786736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:54.786723    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:55.037766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:55.037736    4410 generic.go:356] "Generic (PLEG): container finished" podID="593b2375-c50d-4de0-aca8-57a8df512166" containerID="3d4e9d00e2d6c2185156a1b709c1894cef1fde856f5da36a9af7caa9e9af83fb" exitCode=0
Oct 30 22:29:55.037934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:55.037811    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75" event={"ID":"593b2375-c50d-4de0-aca8-57a8df512166","Type":"ContainerDied","Data":"3d4e9d00e2d6c2185156a1b709c1894cef1fde856f5da36a9af7caa9e9af83fb"}
Oct 30 22:29:56.194824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.194800    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:56.331173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.331102    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\" (UniqueName: \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\") pod \"593b2375-c50d-4de0-aca8-57a8df512166\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") "
Oct 30 22:29:56.331336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.331174    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\") pod \"593b2375-c50d-4de0-aca8-57a8df512166\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") "
Oct 30 22:29:56.331336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.331276    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\") pod \"593b2375-c50d-4de0-aca8-57a8df512166\" (UID: \"593b2375-c50d-4de0-aca8-57a8df512166\") "
Oct 30 22:29:56.331617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.331591    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca" pod "593b2375-c50d-4de0-aca8-57a8df512166" (UID: "593b2375-c50d-4de0-aca8-57a8df512166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:56.331878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.331858    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:56.333501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.333468    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca" pod "593b2375-c50d-4de0-aca8-57a8df512166" (UID: "593b2375-c50d-4de0-aca8-57a8df512166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:56.333737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.333705    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b" pod "593b2375-c50d-4de0-aca8-57a8df512166" (UID: "593b2375-c50d-4de0-aca8-57a8df512166"). InnerVolumeSpecName "kube-api-access-mrl8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:56.406745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.406709    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.433015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.432986    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrl8b\" (UniqueName: \"kubernetes.io/projected/593b2375-c50d-4de0-aca8-57a8df512166-kube-api-access-mrl8b\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:56.433015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:56.433012    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593b2375-c50d-4de0-aca8-57a8df512166-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.027375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.027342    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"]
Oct 30 22:29:57.028017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.027994    4410 cpu_manager.go:414] "containerMap: removing container" podUID="593b2375-c50d-4de0-aca8-57a8df512166" containerName="revoke-keys"
Oct 30 22:29:57.028017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.028015    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593b2375-c50d-4de0-aca8-57a8df512166" containerName="revoke-keys"
Oct 30 22:29:57.028225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.028134    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="593b2375-c50d-4de0-aca8-57a8df512166" containerName="revoke-keys"
Oct 30 22:29:57.031950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.031932    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.052449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.052427    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"
Oct 30 22:29:57.052591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.052420    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75" event={"ID":"593b2375-c50d-4de0-aca8-57a8df512166","Type":"ContainerDied","Data":"f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208"}
Oct 30 22:29:57.052639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.052591    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64946f9df27ae0344d850dd9fc5e9d763523da57de4b1a7d4387f08fac90208"
Oct 30 22:29:57.089481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.089450    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"]
Oct 30 22:29:57.139496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.139455    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.139640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.139515    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdqh\" (UniqueName: \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.139640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.139609    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.240790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.240753    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.241235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.240886    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.241235 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.240923    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdqh\" (UniqueName: \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.241503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.241480    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.243002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.242986    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.297605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.297545    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdqh\" (UniqueName: \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\") pod \"maas-api-revoke-sub-fcc1c26f-lwhwf\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.345903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.345880    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:29:57.417757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.417725    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"]
Oct 30 22:29:57.422726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.422708    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.477734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.477708    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"]
Oct 30 22:29:57.544484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.544458    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpm86\" (UniqueName: \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.544617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.544504    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.544617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.544534    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.646226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.646196    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpm86\" (UniqueName: \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.646368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.646248    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.646439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.646414    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.647065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.647037    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.648662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.648641    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.706114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.706085    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpm86\" (UniqueName: \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\") pod \"maas-api-revoke-sub-fbff259f-lwgtr\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:57.721435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.721406    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"]
Oct 30 22:29:57.721925 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:57.721900    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc464fcba_86e8_4ea4_b18b_43796dea3f4b.slice/crio-7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93 WatchSource:0}: Error finding container 7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93: Status 404 returned error can't find the container with id 7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93
Oct 30 22:29:57.737928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:57.737905    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:29:58.061341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.061311    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf" event={"ID":"c464fcba-86e8-4ea4-b18b-43796dea3f4b","Type":"ContainerStarted","Data":"7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93"}
Oct 30 22:29:58.124927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.124900    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"]
Oct 30 22:29:58.126169 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:58.126139    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cc7018_88b6_4d54_b8c8_6c4322adaa5b.slice/crio-56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7 WatchSource:0}: Error finding container 56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7: Status 404 returned error can't find the container with id 56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7
Oct 30 22:29:58.578492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.578466    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"]
Oct 30 22:29:58.582061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.582044    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.653277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.653241    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"]
Oct 30 22:29:58.760796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.760743    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ws6\" (UniqueName: \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.760933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.760856    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.760991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.760923    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.844572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.844454    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:58.861956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.861927    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ws6\" (UniqueName: \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.862094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.861984    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.862094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.862034    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.862759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.862739    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.864311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.864293    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.918378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.918341    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ws6\" (UniqueName: \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\") pod \"maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:58.945286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:58.945257    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:29:59.075677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.075646    4410 generic.go:356] "Generic (PLEG): container finished" podID="10cc7018-88b6-4d54-b8c8-6c4322adaa5b" containerID="c4c01c736cb129d623d3579d0bed2b53a166223b0d6ba2247e8cb2ec85ba0e76" exitCode=0
Oct 30 22:29:59.075822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.075721    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr" event={"ID":"10cc7018-88b6-4d54-b8c8-6c4322adaa5b","Type":"ContainerDied","Data":"c4c01c736cb129d623d3579d0bed2b53a166223b0d6ba2247e8cb2ec85ba0e76"}
Oct 30 22:29:59.075899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.075820    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr" event={"ID":"10cc7018-88b6-4d54-b8c8-6c4322adaa5b","Type":"ContainerStarted","Data":"56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7"}
Oct 30 22:29:59.077356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.077335    4410 generic.go:356] "Generic (PLEG): container finished" podID="c464fcba-86e8-4ea4-b18b-43796dea3f4b" containerID="683df11f57b8dfac7cc8cdedab4d742c72f3d92f8d74bc23c0d0b51046881b62" exitCode=0
Oct 30 22:29:59.077461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.077417    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf" event={"ID":"c464fcba-86e8-4ea4-b18b-43796dea3f4b","Type":"ContainerDied","Data":"683df11f57b8dfac7cc8cdedab4d742c72f3d92f8d74bc23c0d0b51046881b62"}
Oct 30 22:29:59.331014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:29:59.330987    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"]
Oct 30 22:29:59.332680 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:29:59.332540    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf610e1d3_06c9_4e4a_8300_2117a5c4ec9a.slice/crio-989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f WatchSource:0}: Error finding container 989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f: Status 404 returned error can't find the container with id 989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f
Oct 30 22:30:00.085052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.085014    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt" event={"ID":"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a","Type":"ContainerStarted","Data":"1992052197e1c6bd0f9723228f50b1eaa1de5e0f27105a0b850da6d0a159faef"}
Oct 30 22:30:00.085492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.085063    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt" event={"ID":"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a","Type":"ContainerStarted","Data":"989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f"}
Oct 30 22:30:00.168090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.168056    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:00.172194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.172168    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.191802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.191760    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:00.197845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.196286    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:00.200042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.199629    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.206312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.206289    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:00.207853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.206418    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.211251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.211228    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:00.211332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.211303    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.215650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.215632    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.239659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.239610    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt" podStartSLOduration=2.239594749 podStartE2EDuration="2.239594749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:00.211882682 +0000 UTC m=+2853.300620484" watchObservedRunningTime="2026-10-30 22:30:00.239594749 +0000 UTC m=+2853.328332553"
Oct 30 22:30:00.270540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.270514    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:00.272029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.272010    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:30:00.273659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.273640    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:00.277019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.276996    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:00.279260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.279234    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.279358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.279275    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7792\" (UniqueName: \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.290588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.290570    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:30:00.323633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.323604    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:00.378121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.378021    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:00.380471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380448    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\") pod \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") "
Oct 30 22:30:00.380590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380560    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\") pod \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") "
Oct 30 22:30:00.380670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380590    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\") pod \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") "
Oct 30 22:30:00.380670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380653    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\") pod \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") "
Oct 30 22:30:00.380794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380731    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\" (UniqueName: \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\") pod \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\" (UID: \"10cc7018-88b6-4d54-b8c8-6c4322adaa5b\") "
Oct 30 22:30:00.380901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380874    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca" pod "10cc7018-88b6-4d54-b8c8-6c4322adaa5b" (UID: "10cc7018-88b6-4d54-b8c8-6c4322adaa5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:00.380969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.380883    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\" (UniqueName: \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\") pod \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\" (UID: \"c464fcba-86e8-4ea4-b18b-43796dea3f4b\") "
Oct 30 22:30:00.381195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381158    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.381281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381259    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886kp\" (UniqueName: \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.381339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381301    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb49f\" (UniqueName: \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.381448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381423    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfp8\" (UniqueName: \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.381519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381504    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.381579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381525    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca" pod "c464fcba-86e8-4ea4-b18b-43796dea3f4b" (UID: "c464fcba-86e8-4ea4-b18b-43796dea3f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:00.381579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381530    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.381746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381722    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xztw\" (UniqueName: \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.381892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381762    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.381892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381841    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.381892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381869    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7792\" (UniqueName: \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.382062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.381987    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.382062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.382009    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.382652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.382624    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.385084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.385062    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca" pod "c464fcba-86e8-4ea4-b18b-43796dea3f4b" (UID: "c464fcba-86e8-4ea4-b18b-43796dea3f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:00.385245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.385225    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh" pod "c464fcba-86e8-4ea4-b18b-43796dea3f4b" (UID: "c464fcba-86e8-4ea4-b18b-43796dea3f4b"). InnerVolumeSpecName "kube-api-access-xjdqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:00.385304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.385252    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86" pod "10cc7018-88b6-4d54-b8c8-6c4322adaa5b" (UID: "10cc7018-88b6-4d54-b8c8-6c4322adaa5b"). InnerVolumeSpecName "kube-api-access-fpm86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:00.385304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.385248    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca" pod "10cc7018-88b6-4d54-b8c8-6c4322adaa5b" (UID: "10cc7018-88b6-4d54-b8c8-6c4322adaa5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:00.440561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.440536    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7792\" (UniqueName: \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\") pod \"maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.482784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.482751    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xztw\" (UniqueName: \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.482925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.482836    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.482977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.482965    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.483064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483034    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886kp\" (UniqueName: \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.483125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483078    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb49f\" (UniqueName: \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.483125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483102    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfp8\" (UniqueName: \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.483239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483139    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.483239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483164    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.483358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483249    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdqh\" (UniqueName: \"kubernetes.io/projected/c464fcba-86e8-4ea4-b18b-43796dea3f4b-kube-api-access-xjdqh\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.483358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483266    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c464fcba-86e8-4ea4-b18b-43796dea3f4b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.483358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483281    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.483358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483302    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpm86\" (UniqueName: \"kubernetes.io/projected/10cc7018-88b6-4d54-b8c8-6c4322adaa5b-kube-api-access-fpm86\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.483570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483499    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.483734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483711    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.483920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483899    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.483962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.483945    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.488717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.488701    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:00.540271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.540230    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfp8\" (UniqueName: \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.541916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.541854    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb49f\" (UniqueName: \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\") pod \"maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.542463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.542408    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xztw\" (UniqueName: \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\") pod \"maas-api-key-cleanup-29889990-xtmkb\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.542842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.542753    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-886kp\" (UniqueName: \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\") pod \"maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.551150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.551125    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:00.570635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.570614    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:00.582200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.582176    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:00.589544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.589517    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:00.862249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.861934    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:00.864814 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:30:00.864767    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd2c029_33d7_410a_b640_65ba17c7aeea.slice/crio-2a0ce3edd2f74a53e05ad22bc45f770a5822e4e6e3a022bdf5d83f2c47bef3f9 WatchSource:0}: Error finding container 2a0ce3edd2f74a53e05ad22bc45f770a5822e4e6e3a022bdf5d83f2c47bef3f9: Status 404 returned error can't find the container with id 2a0ce3edd2f74a53e05ad22bc45f770a5822e4e6e3a022bdf5d83f2c47bef3f9
Oct 30 22:30:00.937292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.937269    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:00.937906 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:30:00.937884    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa3b2ca_3f95_41b3_a368_d4f078dfc208.slice/crio-b37810f658668f6bac1ea8ef75c04aade437a810e5d217fcb40c3b684c292a71 WatchSource:0}: Error finding container b37810f658668f6bac1ea8ef75c04aade437a810e5d217fcb40c3b684c292a71: Status 404 returned error can't find the container with id b37810f658668f6bac1ea8ef75c04aade437a810e5d217fcb40c3b684c292a71
Oct 30 22:30:00.973384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:00.973364    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:01.010726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.010696    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:01.036352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.036299    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:01.096909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.096880    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerStarted","Data":"2a0ce3edd2f74a53e05ad22bc45f770a5822e4e6e3a022bdf5d83f2c47bef3f9"}
Oct 30 22:30:01.098281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.098264    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"
Oct 30 22:30:01.098379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.098259    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf" event={"ID":"c464fcba-86e8-4ea4-b18b-43796dea3f4b","Type":"ContainerDied","Data":"7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93"}
Oct 30 22:30:01.098420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.098383    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd29f48e3f4244d3e48f20aaaf63986cfa7cd2ff09dd2fe2684c4a2f257fb93"
Oct 30 22:30:01.099348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.099323    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerStarted","Data":"9c30814e4d4b9821cfa7e040ca324c1c7a7588140b707dbeb6660193246bda4e"}
Oct 30 22:30:01.100668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.100645    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerStarted","Data":"9031bafb00644be7828fc42dc0ff5e947235af1e2a58bc372cfc26c20ee710d0"}
Oct 30 22:30:01.101830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.101797    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerStarted","Data":"b37810f658668f6bac1ea8ef75c04aade437a810e5d217fcb40c3b684c292a71"}
Oct 30 22:30:01.103019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.102997    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerStarted","Data":"645b9d25dd0d834d233de9e709ea26a2851ddbb439aeebb1b7701d90f26dc082"}
Oct 30 22:30:01.104094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.104072    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr" event={"ID":"10cc7018-88b6-4d54-b8c8-6c4322adaa5b","Type":"ContainerDied","Data":"56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7"}
Oct 30 22:30:01.104179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.104092    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"
Oct 30 22:30:01.104179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.104109    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e8828451cf8d9dc7316c297c86a605a3072914b950fc767e04c8dd636313a7"
Oct 30 22:30:01.105513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.105491    4410 generic.go:356] "Generic (PLEG): container finished" podID="f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" containerID="1992052197e1c6bd0f9723228f50b1eaa1de5e0f27105a0b850da6d0a159faef" exitCode=0
Oct 30 22:30:01.105630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:01.105517    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt" event={"ID":"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a","Type":"ContainerDied","Data":"1992052197e1c6bd0f9723228f50b1eaa1de5e0f27105a0b850da6d0a159faef"}
Oct 30 22:30:02.051270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.051238    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"]
Oct 30 22:30:02.054640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.054610    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0265f16f-mkqpb"]
Oct 30 22:30:02.113668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.113638    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerStarted","Data":"d55016ccf657eee303f6cc677dde8ac4df181f21d0878a5da9659a269cea3474"}
Oct 30 22:30:02.115396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.115371    4410 generic.go:356] "Generic (PLEG): container finished" podID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerID="406cf02eaa72f1b43842da056f3634245532ab53209a40d443606d8b82eface4" exitCode=60
Oct 30 22:30:02.115515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.115438    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerDied","Data":"406cf02eaa72f1b43842da056f3634245532ab53209a40d443606d8b82eface4"}
Oct 30 22:30:02.115628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.115615    4410 scope.go:122] "RemoveContainer" containerID="406cf02eaa72f1b43842da056f3634245532ab53209a40d443606d8b82eface4"
Oct 30 22:30:02.117200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.117178    4410 generic.go:356] "Generic (PLEG): container finished" podID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerID="eb9c5a190ab717c592042bb177b498fc722715a3830dd4eefaee5ca71844dba6" exitCode=60
Oct 30 22:30:02.117306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.117288    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerDied","Data":"eb9c5a190ab717c592042bb177b498fc722715a3830dd4eefaee5ca71844dba6"}
Oct 30 22:30:02.117454 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.117441    4410 scope.go:122] "RemoveContainer" containerID="eb9c5a190ab717c592042bb177b498fc722715a3830dd4eefaee5ca71844dba6"
Oct 30 22:30:02.119232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.119206    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerStarted","Data":"82057216ee9b086d921e51fb9c8f9c370b5b14fa08b0ec3bc26eff5e5bb9f59d"}
Oct 30 22:30:02.121033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.121013    4410 generic.go:356] "Generic (PLEG): container finished" podID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerID="185fb0d2ef3bb662bab8571dbf3082716274347a84a556d1c5370eedf8bd9823" exitCode=60
Oct 30 22:30:02.121125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.121096    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerDied","Data":"185fb0d2ef3bb662bab8571dbf3082716274347a84a556d1c5370eedf8bd9823"}
Oct 30 22:30:02.121303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.121283    4410 scope.go:122] "RemoveContainer" containerID="185fb0d2ef3bb662bab8571dbf3082716274347a84a556d1c5370eedf8bd9823"
Oct 30 22:30:02.227939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.227713    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" podStartSLOduration=2.22767677 podStartE2EDuration="2.22767677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.225368327 +0000 UTC m=+2855.314106129" watchObservedRunningTime="2026-10-30 22:30:02.22767677 +0000 UTC m=+2855.316414573"
Oct 30 22:30:02.322675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.322641    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:30:02.507970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.507901    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\" (UniqueName: \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\") pod \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") "
Oct 30 22:30:02.508119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.507972    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\") pod \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") "
Oct 30 22:30:02.508119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.508102    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\") pod \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\" (UID: \"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a\") "
Oct 30 22:30:02.508317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.508295    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca" pod "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" (UID: "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:02.508538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.508512    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.510107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.510082    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca" pod "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" (UID: "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:02.510185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.510116    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6" pod "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" (UID: "f610e1d3-06c9-4e4a-8300-2117a5c4ec9a"). InnerVolumeSpecName "kube-api-access-65ws6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:02.563894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.563842    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" podStartSLOduration=2.56382604 podStartE2EDuration="2.56382604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.560855944 +0000 UTC m=+2855.649593744" watchObservedRunningTime="2026-10-30 22:30:02.56382604 +0000 UTC m=+2855.652563845"
Oct 30 22:30:02.609949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.609923    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.609949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.609947    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65ws6\" (UniqueName: \"kubernetes.io/projected/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a-kube-api-access-65ws6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.917164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:02.917126    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:03.020096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.020056    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0601e5-59cf-4418-a24c-c25f4f3f015b" path="/var/lib/kubelet/pods/4e0601e5-59cf-4418-a24c-c25f4f3f015b/volumes"
Oct 30 22:30:03.137559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.137534    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"
Oct 30 22:30:03.137987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.137529    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt" event={"ID":"f610e1d3-06c9-4e4a-8300-2117a5c4ec9a","Type":"ContainerDied","Data":"989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f"}
Oct 30 22:30:03.137987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.137687    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="989b42b09919f85c07c24976e77bad90fd22a55f47bd71d66ffc7c17505f188f"
Oct 30 22:30:03.139905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.139881    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerStarted","Data":"27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"}
Oct 30 22:30:03.141608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.141588    4410 generic.go:356] "Generic (PLEG): container finished" podID="e871cf88-0d4d-470a-b004-58d361988c5d" containerID="d55016ccf657eee303f6cc677dde8ac4df181f21d0878a5da9659a269cea3474" exitCode=60
Oct 30 22:30:03.141691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.141675    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerDied","Data":"d55016ccf657eee303f6cc677dde8ac4df181f21d0878a5da9659a269cea3474"}
Oct 30 22:30:03.141884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.141869    4410 scope.go:122] "RemoveContainer" containerID="d55016ccf657eee303f6cc677dde8ac4df181f21d0878a5da9659a269cea3474"
Oct 30 22:30:03.143590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.143569    4410 generic.go:356] "Generic (PLEG): container finished" podID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerID="2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0" exitCode=60
Oct 30 22:30:03.143665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.143643    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerDied","Data":"2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0"}
Oct 30 22:30:03.143720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.143703    4410 scope.go:122] "RemoveContainer" containerID="406cf02eaa72f1b43842da056f3634245532ab53209a40d443606d8b82eface4"
Oct 30 22:30:03.144032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.144007    4410 scope.go:122] "RemoveContainer" containerID="2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0"
Oct 30 22:30:03.145035 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:03.144998    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt_odh-ai-gateway-infra(798456d9-d9e5-4192-b832-29fd4f46c7bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb"
Oct 30 22:30:03.147058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.146443    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerStarted","Data":"0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"}
Oct 30 22:30:03.152328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.152294    4410 generic.go:356] "Generic (PLEG): container finished" podID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerID="82057216ee9b086d921e51fb9c8f9c370b5b14fa08b0ec3bc26eff5e5bb9f59d" exitCode=60
Oct 30 22:30:03.152429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.152387    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerDied","Data":"82057216ee9b086d921e51fb9c8f9c370b5b14fa08b0ec3bc26eff5e5bb9f59d"}
Oct 30 22:30:03.152606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.152590    4410 scope.go:122] "RemoveContainer" containerID="82057216ee9b086d921e51fb9c8f9c370b5b14fa08b0ec3bc26eff5e5bb9f59d"
Oct 30 22:30:03.308739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.308693    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" podStartSLOduration=3.308678744 podStartE2EDuration="3.308678744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.30532194 +0000 UTC m=+2856.394059742" watchObservedRunningTime="2026-10-30 22:30:03.308678744 +0000 UTC m=+2856.397416592"
Oct 30 22:30:03.422272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.422200    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" podStartSLOduration=3.422187558 podStartE2EDuration="3.422187558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.41851652 +0000 UTC m=+2856.507254323" watchObservedRunningTime="2026-10-30 22:30:03.422187558 +0000 UTC m=+2856.510925362"
Oct 30 22:30:03.852237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:03.852141    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.161646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.161616    4410 generic.go:356] "Generic (PLEG): container finished" podID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerID="93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f" exitCode=60
Oct 30 22:30:04.162124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.161698    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerDied","Data":"93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f"}
Oct 30 22:30:04.162124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.161757    4410 scope.go:122] "RemoveContainer" containerID="82057216ee9b086d921e51fb9c8f9c370b5b14fa08b0ec3bc26eff5e5bb9f59d"
Oct 30 22:30:04.162124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.162092    4410 scope.go:122] "RemoveContainer" containerID="93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f"
Oct 30 22:30:04.162438 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:04.162415    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-xtmkb_odh-ai-gateway-infra(86ba3cbb-f92b-4d27-b910-4f5f8ea85185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185"
Oct 30 22:30:04.163848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.163827    4410 generic.go:356] "Generic (PLEG): container finished" podID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerID="27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c" exitCode=60
Oct 30 22:30:04.163946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.163911    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerDied","Data":"27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"}
Oct 30 22:30:04.164137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.164123    4410 scope.go:122] "RemoveContainer" containerID="27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"
Oct 30 22:30:04.164403 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:04.164379    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t_odh-ai-gateway-infra(ddd2c029-33d7-410a-b640-65ba17c7aeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea"
Oct 30 22:30:04.165787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.165750    4410 generic.go:356] "Generic (PLEG): container finished" podID="e871cf88-0d4d-470a-b004-58d361988c5d" containerID="9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932" exitCode=60
Oct 30 22:30:04.165896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.165827    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerDied","Data":"9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932"}
Oct 30 22:30:04.166098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.166081    4410 scope.go:122] "RemoveContainer" containerID="9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932"
Oct 30 22:30:04.166343 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:04.166321    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl_odh-ai-gateway-infra(e871cf88-0d4d-470a-b004-58d361988c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" podUID="e871cf88-0d4d-470a-b004-58d361988c5d"
Oct 30 22:30:04.167764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.167750    4410 scope.go:122] "RemoveContainer" containerID="2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0"
Oct 30 22:30:04.168031 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:04.168005    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt_odh-ai-gateway-infra(798456d9-d9e5-4192-b832-29fd4f46c7bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb"
Oct 30 22:30:04.169388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.169365    4410 generic.go:356] "Generic (PLEG): container finished" podID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerID="0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf" exitCode=60
Oct 30 22:30:04.169479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.169446    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerDied","Data":"0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"}
Oct 30 22:30:04.169644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.169630    4410 scope.go:122] "RemoveContainer" containerID="0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"
Oct 30 22:30:04.169878 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:04.169853    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc_odh-ai-gateway-infra(faa3b2ca-3f95-41b3-a368-d4f078dfc208)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208"
Oct 30 22:30:04.174559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.174541    4410 scope.go:122] "RemoveContainer" containerID="185fb0d2ef3bb662bab8571dbf3082716274347a84a556d1c5370eedf8bd9823"
Oct 30 22:30:04.186671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.186652    4410 scope.go:122] "RemoveContainer" containerID="d55016ccf657eee303f6cc677dde8ac4df181f21d0878a5da9659a269cea3474"
Oct 30 22:30:04.198786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.198754    4410 scope.go:122] "RemoveContainer" containerID="eb9c5a190ab717c592042bb177b498fc722715a3830dd4eefaee5ca71844dba6"
Oct 30 22:30:04.787079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.787039    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:04.787079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:04.787057    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:05.178911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:05.178882    4410 scope.go:122] "RemoveContainer" containerID="93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f"
Oct 30 22:30:05.179294 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:05.179140    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-xtmkb_odh-ai-gateway-infra(86ba3cbb-f92b-4d27-b910-4f5f8ea85185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185"
Oct 30 22:30:05.180488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:05.180468    4410 scope.go:122] "RemoveContainer" containerID="27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"
Oct 30 22:30:05.180729 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:05.180705    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t_odh-ai-gateway-infra(ddd2c029-33d7-410a-b640-65ba17c7aeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea"
Oct 30 22:30:05.182041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:05.182022    4410 scope.go:122] "RemoveContainer" containerID="9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932"
Oct 30 22:30:05.182268 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:05.182245    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl_odh-ai-gateway-infra(e871cf88-0d4d-470a-b004-58d361988c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" podUID="e871cf88-0d4d-470a-b004-58d361988c5d"
Oct 30 22:30:05.183549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:05.183530    4410 scope.go:122] "RemoveContainer" containerID="0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"
Oct 30 22:30:05.183722 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:05.183705    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc_odh-ai-gateway-infra(faa3b2ca-3f95-41b3-a368-d4f078dfc208)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208"
Oct 30 22:30:08.325056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.321593    4410 generic.go:356] "Generic (PLEG): container finished" podID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerID="4551b481fa51dc88ca66b0e02e32fe046a56bbfb948ad3964ea2e0c3a7e9afd2" exitCode=0
Oct 30 22:30:08.325056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.321723    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" event={"ID":"5f423c8f-25c4-4402-b660-c0b6adc125a9","Type":"ContainerDied","Data":"4551b481fa51dc88ca66b0e02e32fe046a56bbfb948ad3964ea2e0c3a7e9afd2"}
Oct 30 22:30:08.332092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.326060    4410 generic.go:356] "Generic (PLEG): container finished" podID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerID="86ce50f47d9588534b0133800e8ecf46f08b8cad70f7c6ef6f1247e6caafac5c" exitCode=0
Oct 30 22:30:08.332092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.326171    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" event={"ID":"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe","Type":"ContainerDied","Data":"86ce50f47d9588534b0133800e8ecf46f08b8cad70f7c6ef6f1247e6caafac5c"}
Oct 30 22:30:08.332289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.332242    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:30:08.423805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.416072    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\" (UniqueName: \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\") pod \"5f423c8f-25c4-4402-b660-c0b6adc125a9\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") "
Oct 30 22:30:08.423805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.416253    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\") pod \"5f423c8f-25c4-4402-b660-c0b6adc125a9\" (UID: \"5f423c8f-25c4-4402-b660-c0b6adc125a9\") "
Oct 30 22:30:08.423805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.417410    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume" pod "5f423c8f-25c4-4402-b660-c0b6adc125a9" (UID: "5f423c8f-25c4-4402-b660-c0b6adc125a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:08.428028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.428003    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n" pod "5f423c8f-25c4-4402-b660-c0b6adc125a9" (UID: "5f423c8f-25c4-4402-b660-c0b6adc125a9"). InnerVolumeSpecName "kube-api-access-bb49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:08.441735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.441712    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:30:08.517766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.517667    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f423c8f-25c4-4402-b660-c0b6adc125a9-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.517766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.517705    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb49n\" (UniqueName: \"kubernetes.io/projected/5f423c8f-25c4-4402-b660-c0b6adc125a9-kube-api-access-bb49n\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.619037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.618994    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\") pod \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") "
Oct 30 22:30:08.619225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.619063    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\" (UniqueName: \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\") pod \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\" (UID: \"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe\") "
Oct 30 22:30:08.619489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.619461    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume" pod "69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" (UID: "69ca7788-5a0c-4b1f-b5be-c4a393cf14fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:08.619658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.619637    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.621511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.621481    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws" pod "69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" (UID: "69ca7788-5a0c-4b1f-b5be-c4a393cf14fe"). InnerVolumeSpecName "kube-api-access-p52ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:08.720260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:08.720233    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52ws\" (UniqueName: \"kubernetes.io/projected/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe-kube-api-access-p52ws\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.330620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.330528    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:09.334338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.334318    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"
Oct 30 22:30:09.334475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.334312    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k" event={"ID":"69ca7788-5a0c-4b1f-b5be-c4a393cf14fe","Type":"ContainerDied","Data":"4e5e16c5907ccb030380e06827e49afd05607a3a2544e0ce372d652ee721882a"}
Oct 30 22:30:09.334475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.334441    4410 scope.go:122] "RemoveContainer" containerID="86ce50f47d9588534b0133800e8ecf46f08b8cad70f7c6ef6f1247e6caafac5c"
Oct 30 22:30:09.336001 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.335977    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj" event={"ID":"5f423c8f-25c4-4402-b660-c0b6adc125a9","Type":"ContainerDied","Data":"14e25bf24ca694bbd13171452d085113c608c47f5e0019b9c8bcb059759cb79b"}
Oct 30 22:30:09.336104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.336010    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"
Oct 30 22:30:09.344396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.344378    4410 scope.go:122] "RemoveContainer" containerID="4551b481fa51dc88ca66b0e02e32fe046a56bbfb948ad3964ea2e0c3a7e9afd2"
Oct 30 22:30:09.498510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.498480    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:30:09.508964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.502804    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-086575-6f88b54769-9rpxj"]
Oct 30 22:30:09.655819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.655767    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:30:09.658759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:09.658738    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-086575-6c7cc78fcf-lxr8k"]
Oct 30 22:30:10.709436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:10.709396    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:10.871665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:10.871623    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.019905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:11.019825    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" path="/var/lib/kubelet/pods/5f423c8f-25c4-4402-b660-c0b6adc125a9/volumes"
Oct 30 22:30:11.020321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:11.020298    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" path="/var/lib/kubelet/pods/69ca7788-5a0c-4b1f-b5be-c4a393cf14fe/volumes"
Oct 30 22:30:14.015345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.015314    4410 scope.go:122] "RemoveContainer" containerID="2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0"
Oct 30 22:30:14.015763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.015434    4410 scope.go:122] "RemoveContainer" containerID="27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"
Oct 30 22:30:14.084267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.084243    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.250431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.250403    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.374049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.374016    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerStarted","Data":"fe0595e90e5698201dfd85b2802c5533507a560e7f7721848b8275891edb32e1"}
Oct 30 22:30:14.413717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.413683    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.524333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:14.524286    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" podStartSLOduration=14.524272783 podStartE2EDuration="14.524272783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:14.522720252 +0000 UTC m=+2867.611458056" watchObservedRunningTime="2026-10-30 22:30:14.524272783 +0000 UTC m=+2867.613010582"
Oct 30 22:30:15.382091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.382002    4410 generic.go:356] "Generic (PLEG): container finished" podID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerID="fe0595e90e5698201dfd85b2802c5533507a560e7f7721848b8275891edb32e1" exitCode=60
Oct 30 22:30:15.382091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.382069    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerDied","Data":"fe0595e90e5698201dfd85b2802c5533507a560e7f7721848b8275891edb32e1"}
Oct 30 22:30:15.382744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.382133    4410 scope.go:122] "RemoveContainer" containerID="2ebc1c908050b0b6ec8603a2a65f1cccc6a944ee23ba1361c223c2d61e4c4ef0"
Oct 30 22:30:15.382744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.382484    4410 scope.go:122] "RemoveContainer" containerID="fe0595e90e5698201dfd85b2802c5533507a560e7f7721848b8275891edb32e1"
Oct 30 22:30:15.382891 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:15.382823    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt_odh-ai-gateway-infra(798456d9-d9e5-4192-b832-29fd4f46c7bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb"
Oct 30 22:30:15.384274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.384253    4410 generic.go:356] "Generic (PLEG): container finished" podID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerID="a8f39d4cdbce30febe948487975fbfebcbfbbaef2d58dd815d7ab1d9c8069e62" exitCode=60
Oct 30 22:30:15.384368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.384291    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerDied","Data":"a8f39d4cdbce30febe948487975fbfebcbfbbaef2d58dd815d7ab1d9c8069e62"}
Oct 30 22:30:15.384621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.384606    4410 scope.go:122] "RemoveContainer" containerID="a8f39d4cdbce30febe948487975fbfebcbfbbaef2d58dd815d7ab1d9c8069e62"
Oct 30 22:30:15.384890 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:15.384869    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t_odh-ai-gateway-infra(ddd2c029-33d7-410a-b640-65ba17c7aeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea"
Oct 30 22:30:15.394970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.394953    4410 scope.go:122] "RemoveContainer" containerID="27cafcbfacbc7646602c0d1a9b689d0160ac82e44c01ef7be886d5998cd63e9c"
Oct 30 22:30:15.533595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:15.533565    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:16.012465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.012432    4410 scope.go:122] "RemoveContainer" containerID="9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932"
Oct 30 22:30:16.012750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.012716    4410 scope.go:122] "RemoveContainer" containerID="0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"
Oct 30 22:30:16.404756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.404722    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerStarted","Data":"a80b8811f83bd155df5239b415bdb2dd0069cca9ed57e7b46837cd57ed7788bf"}
Oct 30 22:30:16.564613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.564590    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:16.628083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.628012    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:16.716541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.716510    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\" (UniqueName: \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\") pod \"798456d9-d9e5-4192-b832-29fd4f46c7bb\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") "
Oct 30 22:30:16.716702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.716597    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\") pod \"798456d9-d9e5-4192-b832-29fd4f46c7bb\" (UID: \"798456d9-d9e5-4192-b832-29fd4f46c7bb\") "
Oct 30 22:30:16.717283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.717251    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca" pod "798456d9-d9e5-4192-b832-29fd4f46c7bb" (UID: "798456d9-d9e5-4192-b832-29fd4f46c7bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.718784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.718740    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f" pod "798456d9-d9e5-4192-b832-29fd4f46c7bb" (UID: "798456d9-d9e5-4192-b832-29fd4f46c7bb"). InnerVolumeSpecName "kube-api-access-wb49f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.783110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.783043    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:16.818324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.818294    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb49f\" (UniqueName: \"kubernetes.io/projected/798456d9-d9e5-4192-b832-29fd4f46c7bb-kube-api-access-wb49f\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.818324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.818323    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798456d9-d9e5-4192-b832-29fd4f46c7bb-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.918817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.918787    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\") pod \"ddd2c029-33d7-410a-b640-65ba17c7aeea\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") "
Oct 30 22:30:16.918984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.918896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\" (UniqueName: \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\") pod \"ddd2c029-33d7-410a-b640-65ba17c7aeea\" (UID: \"ddd2c029-33d7-410a-b640-65ba17c7aeea\") "
Oct 30 22:30:16.919103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.919083    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca" pod "ddd2c029-33d7-410a-b640-65ba17c7aeea" (UID: "ddd2c029-33d7-410a-b640-65ba17c7aeea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.919270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.919258    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2c029-33d7-410a-b640-65ba17c7aeea-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.920852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:16.920832    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792" pod "ddd2c029-33d7-410a-b640-65ba17c7aeea" (UID: "ddd2c029-33d7-410a-b640-65ba17c7aeea"). InnerVolumeSpecName "kube-api-access-m7792". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.016211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.016181    4410 scope.go:122] "RemoveContainer" containerID="93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f"
Oct 30 22:30:17.027829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.021444    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7792\" (UniqueName: \"kubernetes.io/projected/ddd2c029-33d7-410a-b640-65ba17c7aeea-kube-api-access-m7792\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.414311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.414278    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerStarted","Data":"3ef966aafdeac850b0285e60039dc82ef4f5afd40d4e4c80a34a3b301a212a73"}
Oct 30 22:30:17.415921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.415885    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"
Oct 30 22:30:17.416076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.415914    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t" event={"ID":"ddd2c029-33d7-410a-b640-65ba17c7aeea","Type":"ContainerDied","Data":"2a0ce3edd2f74a53e05ad22bc45f770a5822e4e6e3a022bdf5d83f2c47bef3f9"}
Oct 30 22:30:17.416076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.415973    4410 scope.go:122] "RemoveContainer" containerID="a8f39d4cdbce30febe948487975fbfebcbfbbaef2d58dd815d7ab1d9c8069e62"
Oct 30 22:30:17.418527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.418486    4410 generic.go:356] "Generic (PLEG): container finished" podID="e871cf88-0d4d-470a-b004-58d361988c5d" containerID="422a74b809cb5413b85f77a0f046418e9a65d0da30715015f8b983e013aed1ff" exitCode=60
Oct 30 22:30:17.418653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.418558    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerDied","Data":"422a74b809cb5413b85f77a0f046418e9a65d0da30715015f8b983e013aed1ff"}
Oct 30 22:30:17.419032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.419013    4410 scope.go:122] "RemoveContainer" containerID="422a74b809cb5413b85f77a0f046418e9a65d0da30715015f8b983e013aed1ff"
Oct 30 22:30:17.419335 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:17.419309    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl_odh-ai-gateway-infra(e871cf88-0d4d-470a-b004-58d361988c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" podUID="e871cf88-0d4d-470a-b004-58d361988c5d"
Oct 30 22:30:17.421545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.420328    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"
Oct 30 22:30:17.421545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.420813    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt" event={"ID":"798456d9-d9e5-4192-b832-29fd4f46c7bb","Type":"ContainerDied","Data":"9031bafb00644be7828fc42dc0ff5e947235af1e2a58bc372cfc26c20ee710d0"}
Oct 30 22:30:17.423098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.423074    4410 generic.go:356] "Generic (PLEG): container finished" podID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerID="a80b8811f83bd155df5239b415bdb2dd0069cca9ed57e7b46837cd57ed7788bf" exitCode=60
Oct 30 22:30:17.423260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.423122    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerDied","Data":"a80b8811f83bd155df5239b415bdb2dd0069cca9ed57e7b46837cd57ed7788bf"}
Oct 30 22:30:17.423459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.423444    4410 scope.go:122] "RemoveContainer" containerID="a80b8811f83bd155df5239b415bdb2dd0069cca9ed57e7b46837cd57ed7788bf"
Oct 30 22:30:17.423708 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:17.423688    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc_odh-ai-gateway-infra(faa3b2ca-3f95-41b3-a368-d4f078dfc208)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208"
Oct 30 22:30:17.428233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.428207    4410 scope.go:122] "RemoveContainer" containerID="9068b649d2d84d04725f0e94721090bebbd9a9bd893cee0d0089675dad78a932"
Oct 30 22:30:17.442624 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.442605    4410 scope.go:122] "RemoveContainer" containerID="fe0595e90e5698201dfd85b2802c5533507a560e7f7721848b8275891edb32e1"
Oct 30 22:30:17.452395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.452375    4410 scope.go:122] "RemoveContainer" containerID="0193ca1aea06ac935603cb999f6422ce1b9382f0f18ed8fce0e34282544f85bf"
Oct 30 22:30:17.642732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.642695    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:17.647810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.647759    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b563fd-29889990-2f9dt"]
Oct 30 22:30:17.653685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.653658    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:17.870206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.870180    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:17.878698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:17.878664    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-42e8a1-29889990-pmw7t"]
Oct 30 22:30:18.430969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.430933    4410 generic.go:356] "Generic (PLEG): container finished" podID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerID="3ef966aafdeac850b0285e60039dc82ef4f5afd40d4e4c80a34a3b301a212a73" exitCode=60
Oct 30 22:30:18.431437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.431008    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerDied","Data":"3ef966aafdeac850b0285e60039dc82ef4f5afd40d4e4c80a34a3b301a212a73"}
Oct 30 22:30:18.431437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.431070    4410 scope.go:122] "RemoveContainer" containerID="93d4f69b27950b6b6c0105c9ba26f4fa683a49e41e19b9f39b475b9be3bb921f"
Oct 30 22:30:18.431437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.431436    4410 scope.go:122] "RemoveContainer" containerID="3ef966aafdeac850b0285e60039dc82ef4f5afd40d4e4c80a34a3b301a212a73"
Oct 30 22:30:18.431726 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:18.431697    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-xtmkb_odh-ai-gateway-infra(86ba3cbb-f92b-4d27-b910-4f5f8ea85185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185"
Oct 30 22:30:18.537723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.537692    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:18.590026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.590002    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:18.741479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.741409    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\") pod \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") "
Oct 30 22:30:18.741479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.741473    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\" (UniqueName: \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\") pod \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\" (UID: \"faa3b2ca-3f95-41b3-a368-d4f078dfc208\") "
Oct 30 22:30:18.741842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.741816    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca" pod "faa3b2ca-3f95-41b3-a368-d4f078dfc208" (UID: "faa3b2ca-3f95-41b3-a368-d4f078dfc208"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.742054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.742035    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa3b2ca-3f95-41b3-a368-d4f078dfc208-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.743452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.743429    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8" pod "faa3b2ca-3f95-41b3-a368-d4f078dfc208" (UID: "faa3b2ca-3f95-41b3-a368-d4f078dfc208"). InnerVolumeSpecName "kube-api-access-wpfp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.842905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:18.842871    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfp8\" (UniqueName: \"kubernetes.io/projected/faa3b2ca-3f95-41b3-a368-d4f078dfc208-kube-api-access-wpfp8\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.001595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.001504    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:19.019394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.019361    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" path="/var/lib/kubelet/pods/798456d9-d9e5-4192-b832-29fd4f46c7bb/volumes"
Oct 30 22:30:19.019813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.019795    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" path="/var/lib/kubelet/pods/ddd2c029-33d7-410a-b640-65ba17c7aeea/volumes"
Oct 30 22:30:19.188655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.188623    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:19.238209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.238168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:19.347416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.347330    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\" (UniqueName: \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\") pod \"e871cf88-0d4d-470a-b004-58d361988c5d\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") "
Oct 30 22:30:19.347592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.347416    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\") pod \"e871cf88-0d4d-470a-b004-58d361988c5d\" (UID: \"e871cf88-0d4d-470a-b004-58d361988c5d\") "
Oct 30 22:30:19.347755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.347731    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca" pod "e871cf88-0d4d-470a-b004-58d361988c5d" (UID: "e871cf88-0d4d-470a-b004-58d361988c5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.347887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.347871    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e871cf88-0d4d-470a-b004-58d361988c5d-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.349439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.349416    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp" pod "e871cf88-0d4d-470a-b004-58d361988c5d" (UID: "e871cf88-0d4d-470a-b004-58d361988c5d"). InnerVolumeSpecName "kube-api-access-886kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.445304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.445267    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"
Oct 30 22:30:19.445304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.445283    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl" event={"ID":"e871cf88-0d4d-470a-b004-58d361988c5d","Type":"ContainerDied","Data":"9c30814e4d4b9821cfa7e040ca324c1c7a7588140b707dbeb6660193246bda4e"}
Oct 30 22:30:19.445881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.445370    4410 scope.go:122] "RemoveContainer" containerID="422a74b809cb5413b85f77a0f046418e9a65d0da30715015f8b983e013aed1ff"
Oct 30 22:30:19.446899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.446878    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"
Oct 30 22:30:19.447026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.446893    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc" event={"ID":"faa3b2ca-3f95-41b3-a368-d4f078dfc208","Type":"ContainerDied","Data":"b37810f658668f6bac1ea8ef75c04aade437a810e5d217fcb40c3b684c292a71"}
Oct 30 22:30:19.448607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.448582    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886kp\" (UniqueName: \"kubernetes.io/projected/e871cf88-0d4d-470a-b004-58d361988c5d-kube-api-access-886kp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.457736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.457719    4410 scope.go:122] "RemoveContainer" containerID="a80b8811f83bd155df5239b415bdb2dd0069cca9ed57e7b46837cd57ed7788bf"
Oct 30 22:30:19.589888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.589866    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:19.610741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.610681    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:19.615534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.615514    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29889990-ml5wc"]
Oct 30 22:30:19.722502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.722462    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:19.732088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.732058    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1c3eef-29889990-v9pgl"]
Oct 30 22:30:19.751043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.751004    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\" (UniqueName: \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\") pod \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") "
Oct 30 22:30:19.751246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.751230    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\") pod \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\" (UID: \"86ba3cbb-f92b-4d27-b910-4f5f8ea85185\") "
Oct 30 22:30:19.751580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.751558    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca" pod "86ba3cbb-f92b-4d27-b910-4f5f8ea85185" (UID: "86ba3cbb-f92b-4d27-b910-4f5f8ea85185"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.753018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.752993    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw" pod "86ba3cbb-f92b-4d27-b910-4f5f8ea85185" (UID: "86ba3cbb-f92b-4d27-b910-4f5f8ea85185"). InnerVolumeSpecName "kube-api-access-5xztw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.853240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.853201    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.853240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:19.853232    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xztw\" (UniqueName: \"kubernetes.io/projected/86ba3cbb-f92b-4d27-b910-4f5f8ea85185-kube-api-access-5xztw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.463004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:20.462977    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"
Oct 30 22:30:20.463418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:20.462973    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb" event={"ID":"86ba3cbb-f92b-4d27-b910-4f5f8ea85185","Type":"ContainerDied","Data":"645b9d25dd0d834d233de9e709ea26a2851ddbb439aeebb1b7701d90f26dc082"}
Oct 30 22:30:20.463418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:20.463099    4410 scope.go:122] "RemoveContainer" containerID="3ef966aafdeac850b0285e60039dc82ef4f5afd40d4e4c80a34a3b301a212a73"
Oct 30 22:30:20.521923 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:20.521894    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:30:20.627380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:20.627352    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:20.630398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:20.630372    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-xtmkb"]
Oct 30 22:30:21.018903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:21.018870    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" path="/var/lib/kubelet/pods/86ba3cbb-f92b-4d27-b910-4f5f8ea85185/volumes"
Oct 30 22:30:21.019313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:21.019299    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" path="/var/lib/kubelet/pods/e871cf88-0d4d-470a-b004-58d361988c5d/volumes"
Oct 30 22:30:21.019657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:21.019645    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" path="/var/lib/kubelet/pods/faa3b2ca-3f95-41b3-a368-d4f078dfc208/volumes"
Oct 30 22:30:21.046361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:21.046338    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"]
Oct 30 22:30:21.051486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:21.051462    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca986360-z6sss"]
Oct 30 22:30:22.773526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773495    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"]
Oct 30 22:30:22.773963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773949    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.774014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773966    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.774014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773981    4410 cpu_manager.go:414] "containerMap: removing container" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.774014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773987    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.774014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.773999    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" containerName="revoke-keys"
Oct 30 22:30:22.774014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774005    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" containerName="revoke-keys"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774018    4410 cpu_manager.go:414] "containerMap: removing container" podUID="10cc7018-88b6-4d54-b8c8-6c4322adaa5b" containerName="revoke-keys"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774023    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cc7018-88b6-4d54-b8c8-6c4322adaa5b" containerName="revoke-keys"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774037    4410 cpu_manager.go:414] "containerMap: removing container" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774044    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774052    4410 cpu_manager.go:414] "containerMap: removing container" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774056    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774066    4410 cpu_manager.go:414] "containerMap: removing container" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774070    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774078    4410 cpu_manager.go:414] "containerMap: removing container" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774082    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774089    4410 cpu_manager.go:414] "containerMap: removing container" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774094    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774100    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774104    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774115    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774119    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774125    4410 cpu_manager.go:414] "containerMap: removing container" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774129    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774136    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774140    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774146    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774150    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774156    4410 cpu_manager.go:414] "containerMap: removing container" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774161    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774172    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c464fcba-86e8-4ea4-b18b-43796dea3f4b" containerName="revoke-keys"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774178    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c464fcba-86e8-4ea4-b18b-43796dea3f4b" containerName="revoke-keys"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774186    4410 cpu_manager.go:414] "containerMap: removing container" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.774181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774190    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774196    4410 cpu_manager.go:414] "containerMap: removing container" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774201    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774264    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774271    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774278    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ca7788-5a0c-4b1f-b5be-c4a393cf14fe" containerName="payload-pre-processing"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774284    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774290    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774295    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774302    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774307    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" containerName="revoke-keys"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774312    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774317    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f423c8f-25c4-4402-b660-c0b6adc125a9" containerName="payload-processing"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774323    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c464fcba-86e8-4ea4-b18b-43796dea3f4b" containerName="revoke-keys"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774327    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774333    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774338    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cc7018-88b6-4d54-b8c8-6c4322adaa5b" containerName="revoke-keys"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774343    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:22.775032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.774348    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:22.779027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.779010    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.834432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.834396    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"]
Oct 30 22:30:22.879693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.879666    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:22.879925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.879907    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:30:22.884384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.884364    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.884516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.884499    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.884571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.884558    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zq5j\" (UniqueName: \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.928380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.928357    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:22.928499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.928383    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:22.985807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.985769    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.985915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.985836    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zq5j\" (UniqueName: \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.985915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.985889    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.986394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.986378    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:22.988089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:22.988052    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:23.017849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.017769    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:23.020445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.020422    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62715d1d-70a7-439e-b281-50ba9f9c646d" path="/var/lib/kubelet/pods/62715d1d-70a7-439e-b281-50ba9f9c646d/volumes"
Oct 30 22:30:23.042661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.042605    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zq5j\" (UniqueName: \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\") pod \"maas-api-revoke-sub-fcf04f3f-d82lz\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:23.094028 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.094004    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:23.385031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.384948    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:23.480136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.480102    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"]
Oct 30 22:30:23.495864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.495821    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz" event={"ID":"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c","Type":"ContainerStarted","Data":"db7b739b9ab8e4e844d0c6ef08a6007969c845217d59e705677d5f74f6e3f216"}
Oct 30 22:30:23.593385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.593352    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:23.782421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:23.782365    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:24.505332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:24.505297    4410 generic.go:356] "Generic (PLEG): container finished" podID="3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" containerID="9487450bd534667a3ef8de530e170eebf0c56f1ced14c49d879713b37dd143b5" exitCode=0
Oct 30 22:30:24.505484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:24.505359    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz" event={"ID":"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c","Type":"ContainerDied","Data":"9487450bd534667a3ef8de530e170eebf0c56f1ced14c49d879713b37dd143b5"}
Oct 30 22:30:24.958867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:24.958821    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:25.178512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.178477    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:25.338752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.338669    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:25.445151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.445119    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"]
Oct 30 22:30:25.494212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.494180    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a853fe-q4tgt"]
Oct 30 22:30:25.658152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.658125    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:25.714175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.714152    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\") pod \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") "
Oct 30 22:30:25.714313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.714213    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\" (UniqueName: \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\") pod \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") "
Oct 30 22:30:25.714384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.714358    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\") pod \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\" (UID: \"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c\") "
Oct 30 22:30:25.714712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.714683    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca" pod "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" (UID: "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:25.716199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.716179    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca" pod "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" (UID: "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:25.716352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.716334    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j" pod "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" (UID: "3c39f1e6-7eae-485d-9aa6-fe325afe7c2c"). InnerVolumeSpecName "kube-api-access-2zq5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.815689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.815658    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.815689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.815690    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.815904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.815700    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zq5j\" (UniqueName: \"kubernetes.io/projected/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c-kube-api-access-2zq5j\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.861240 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:25.861210    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.520450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:26.520420    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"
Oct 30 22:30:26.520868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:26.520463    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz" event={"ID":"3c39f1e6-7eae-485d-9aa6-fe325afe7c2c","Type":"ContainerDied","Data":"db7b739b9ab8e4e844d0c6ef08a6007969c845217d59e705677d5f74f6e3f216"}
Oct 30 22:30:26.520868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:26.520516    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7b739b9ab8e4e844d0c6ef08a6007969c845217d59e705677d5f74f6e3f216"
Oct 30 22:30:27.020054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:27.020020    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f610e1d3-06c9-4e4a-8300-2117a5c4ec9a" path="/var/lib/kubelet/pods/f610e1d3-06c9-4e4a-8300-2117a5c4ec9a/volumes"
Oct 30 22:30:28.825687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:28.825653    4410 scope.go:122] "RemoveContainer" containerID="31fad34cc0a9108442311948a76f27de5e35001c1f644945124be937ae68643b"
Oct 30 22:30:30.051238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:30.051205    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:30.372607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:30.372516    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:32.183271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:32.183233    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:32.398147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:32.398109    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:32.570944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:32.570843    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.356611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.356573    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"]
Oct 30 22:30:33.357027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357011    4410 cpu_manager.go:414] "containerMap: removing container" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:33.357070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357028    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798456d9-d9e5-4192-b832-29fd4f46c7bb" containerName="cleanup"
Oct 30 22:30:33.357070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357049    4410 cpu_manager.go:414] "containerMap: removing container" podUID="3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" containerName="revoke-keys"
Oct 30 22:30:33.357070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357054    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" containerName="revoke-keys"
Oct 30 22:30:33.357170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357074    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:33.357170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357079    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:33.357170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357151    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" containerName="revoke-keys"
Oct 30 22:30:33.357170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357160    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ba3cbb-f92b-4d27-b910-4f5f8ea85185" containerName="cleanup"
Oct 30 22:30:33.357170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357168    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa3b2ca-3f95-41b3-a368-d4f078dfc208" containerName="cleanup"
Oct 30 22:30:33.357319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357176    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2c029-33d7-410a-b640-65ba17c7aeea" containerName="cleanup"
Oct 30 22:30:33.357319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.357182    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e871cf88-0d4d-470a-b004-58d361988c5d" containerName="cleanup"
Oct 30 22:30:33.362230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.362213    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.417067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.417041    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"]
Oct 30 22:30:33.462427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.462400    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:30:33.462559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.462430    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:33.462559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.462479    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:33.463231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.463216    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:33.492947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.492921    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.493050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.492983    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.493101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.493055    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qpj\" (UniqueName: \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.593852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.593816    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.594052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.593922    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.594052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.593942    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qpj\" (UniqueName: \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.594600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.594577    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.596210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.596185    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.651553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.651522    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qpj\" (UniqueName: \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\") pod \"maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:33.675960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:33.675929    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:34.052121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:34.052095    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"]
Oct 30 22:30:34.053697 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:30:34.053668    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189f79fd_272c_4c40_bea8_a308ebb96f29.slice/crio-ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e WatchSource:0}: Error finding container ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e: Status 404 returned error can't find the container with id ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e
Oct 30 22:30:34.386602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:34.386575    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:34.576090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:34.576057    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l" event={"ID":"189f79fd-272c-4c40-bea8-a308ebb96f29","Type":"ContainerStarted","Data":"829e2663facce2630e23b36ed6589f8326ea076226a6f6410c95d34dc7262582"}
Oct 30 22:30:34.576225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:34.576122    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l" event={"ID":"189f79fd-272c-4c40-bea8-a308ebb96f29","Type":"ContainerStarted","Data":"ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e"}
Oct 30 22:30:34.695398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:34.695350    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l" podStartSLOduration=1.695334961 podStartE2EDuration="1.695334961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:34.691622897 +0000 UTC m=+2887.780360717" watchObservedRunningTime="2026-10-30 22:30:34.695334961 +0000 UTC m=+2887.784072765"
Oct 30 22:30:35.369207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:35.369036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.607798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:35.606837    4410 generic.go:356] "Generic (PLEG): container finished" podID="189f79fd-272c-4c40-bea8-a308ebb96f29" containerID="829e2663facce2630e23b36ed6589f8326ea076226a6f6410c95d34dc7262582" exitCode=0
Oct 30 22:30:35.607798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:35.606926    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l" event={"ID":"189f79fd-272c-4c40-bea8-a308ebb96f29","Type":"ContainerDied","Data":"829e2663facce2630e23b36ed6589f8326ea076226a6f6410c95d34dc7262582"}
Oct 30 22:30:35.665808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:35.665573    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.940854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:35.938814    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.176801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.175927    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:30:36.176801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.176013    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:36.763696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.763670    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:36.849278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.849251    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\" (UniqueName: \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\") pod \"189f79fd-272c-4c40-bea8-a308ebb96f29\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") "
Oct 30 22:30:36.849411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.849304    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\") pod \"189f79fd-272c-4c40-bea8-a308ebb96f29\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") "
Oct 30 22:30:36.849411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.849402    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\") pod \"189f79fd-272c-4c40-bea8-a308ebb96f29\" (UID: \"189f79fd-272c-4c40-bea8-a308ebb96f29\") "
Oct 30 22:30:36.849810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.849760    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca" pod "189f79fd-272c-4c40-bea8-a308ebb96f29" (UID: "189f79fd-272c-4c40-bea8-a308ebb96f29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:36.851332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.851314    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca" pod "189f79fd-272c-4c40-bea8-a308ebb96f29" (UID: "189f79fd-272c-4c40-bea8-a308ebb96f29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:36.851414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.851388    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj" pod "189f79fd-272c-4c40-bea8-a308ebb96f29" (UID: "189f79fd-272c-4c40-bea8-a308ebb96f29"). InnerVolumeSpecName "kube-api-access-r6qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:36.950836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.950802    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:36.950836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.950830    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6qpj\" (UniqueName: \"kubernetes.io/projected/189f79fd-272c-4c40-bea8-a308ebb96f29-kube-api-access-r6qpj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:36.950836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:36.950841    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189f79fd-272c-4c40-bea8-a308ebb96f29-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.591051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.591016    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.621646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.621618    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"
Oct 30 22:30:37.621766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.621657    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l" event={"ID":"189f79fd-272c-4c40-bea8-a308ebb96f29","Type":"ContainerDied","Data":"ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e"}
Oct 30 22:30:37.621766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.621705    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef318afc39ae270c8e00e93b867068e2d31e534c1ddf5f95a263da91c9b2ea1e"
Oct 30 22:30:37.794256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.794221    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.979682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:37.979647    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.109864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:41.109833    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.086608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.086578    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.244642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.244609    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.398166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.398131    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.543061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.543027    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:30:42.543456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.543443    4410 cpu_manager.go:414] "containerMap: removing container" podUID="189f79fd-272c-4c40-bea8-a308ebb96f29" containerName="revoke-keys"
Oct 30 22:30:42.543502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.543458    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189f79fd-272c-4c40-bea8-a308ebb96f29" containerName="revoke-keys"
Oct 30 22:30:42.543566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.543557    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="189f79fd-272c-4c40-bea8-a308ebb96f29" containerName="revoke-keys"
Oct 30 22:30:42.547044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.547022    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.600663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.600632    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a0116e\""
Oct 30 22:30:42.601517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.600969    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a0116e-dockercfg-flgt4\""
Oct 30 22:30:42.612074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.612031    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:30:42.614207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.614172    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmctg\" (UniqueName: \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.614395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.614373    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.715562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.715484    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmctg\" (UniqueName: \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.715562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.715519    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.716207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.716183    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.772688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.772653    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmctg\" (UniqueName: \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\") pod \"payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:42.859783 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:42.859748    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:43.229707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:43.229678    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:30:43.231105 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:30:43.231075    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d111bba_f68c_4ca3_9fc8_21c3d5b80eaf.slice/crio-705c055be9ca260d72539ac3050180e65639214186818ce9ca3d460bfd6546fc WatchSource:0}: Error finding container 705c055be9ca260d72539ac3050180e65639214186818ce9ca3d460bfd6546fc: Status 404 returned error can't find the container with id 705c055be9ca260d72539ac3050180e65639214186818ce9ca3d460bfd6546fc
Oct 30 22:30:43.667964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:43.667931    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" event={"ID":"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf","Type":"ContainerStarted","Data":"4312a5fd83b5a495b18305edf15f2066de2d4668b118c3b890696385d7bbb0a7"}
Oct 30 22:30:43.668359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:43.667992    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" event={"ID":"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf","Type":"ContainerStarted","Data":"705c055be9ca260d72539ac3050180e65639214186818ce9ca3d460bfd6546fc"}
Oct 30 22:30:43.668359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:43.668020    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:43.780149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:43.780100    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" podStartSLOduration=1.78008691 podStartE2EDuration="1.78008691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:43.778719279 +0000 UTC m=+2896.867457084" watchObservedRunningTime="2026-10-30 22:30:43.78008691 +0000 UTC m=+2896.868824774"
Oct 30 22:30:46.001206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:46.001160    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.198922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:46.198885    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.459447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:46.459404    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.052821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:47.052790    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"]
Oct 30 22:30:47.102143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:47.102115    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ffadc-g6qzm"]
Oct 30 22:30:49.019491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.019454    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05497d8-b999-4ba1-9449-79676d5d3da1" path="/var/lib/kubelet/pods/d05497d8-b999-4ba1-9449-79676d5d3da1/volumes"
Oct 30 22:30:49.818144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.818109    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:30:49.822119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.822101    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.874588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.874562    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b8717e-openshift-default-dockercfg-pqzrg\""
Oct 30 22:30:49.896168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896139    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdb2\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896174    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896302    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896331    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896354    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896450    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896476    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896563    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.896616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.896593    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.927452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.927426    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:30:49.997827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.997790    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.997844    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.997954    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdb2\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.997995    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998121    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998157    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998187    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998214    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998270    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998308    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998504    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998585    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998700    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:49.998908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:49.998864    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.000720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.000695    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.001061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.001036    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.054582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.054559    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.054916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.054645    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdb2\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.134612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.134547    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:50.521242 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:30:50.521207    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:30:50.585651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.585620    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:30:50.586007 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:30:50.585976    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90409607_8a45_4049_94ac_49e394950d33.slice/crio-29e2eb71647c765221442d774eb56c37ebae97c84e5c28c75033e8353e84f7b1 WatchSource:0}: Error finding container 29e2eb71647c765221442d774eb56c37ebae97c84e5c28c75033e8353e84f7b1: Status 404 returned error can't find the container with id 29e2eb71647c765221442d774eb56c37ebae97c84e5c28c75033e8353e84f7b1
Oct 30 22:30:50.588306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.588279    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:30:50.588393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.588337    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:30:50.588393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.588363    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:30:50.717159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.717115    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" event={"ID":"90409607-8a45-4049-94ac-49e394950d33","Type":"ContainerStarted","Data":"1f7c34992361300f6a826f2316f62cf2147d8de4557ff9c49ddf5e92aa30bf19"}
Oct 30 22:30:50.717334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.717177    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" event={"ID":"90409607-8a45-4049-94ac-49e394950d33","Type":"ContainerStarted","Data":"29e2eb71647c765221442d774eb56c37ebae97c84e5c28c75033e8353e84f7b1"}
Oct 30 22:30:50.839984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:50.839861    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podStartSLOduration=1.8398419160000001 podStartE2EDuration="1.839841916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:50.836520936 +0000 UTC m=+2903.925258745" watchObservedRunningTime="2026-10-30 22:30:50.839841916 +0000 UTC m=+2903.928579722"
Oct 30 22:30:51.114553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:51.114458    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:51.135521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:51.135488    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:51.137488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:51.137458    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:51.137637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:51.137510    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:52.068058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:52.068013    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:52.135818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:52.135770    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:52.136353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:52.135839    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:52.218652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:52.218616    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:52.831816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:52.831782    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.135274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.135189    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:53.135274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.135250    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:53.171237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.171193    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.335268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.335215    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.372794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.372746    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"]
Oct 30 22:30:53.419875 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.419844    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-086575-d4sfj"]
Oct 30 22:30:53.766852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:53.766118    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.135812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:54.135719    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:54.135812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:54.135797    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:54.678196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:54.678167    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:30:55.020289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:55.020204    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6966b15-4705-41ec-b43d-294724484249" path="/var/lib/kubelet/pods/e6966b15-4705-41ec-b43d-294724484249/volumes"
Oct 30 22:30:55.135005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:55.134961    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:55.135180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:55.135028    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:56.135625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:56.135592    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:56.136040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:56.135652    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:57.135130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:57.135097    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:30:57.135342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:57.135156    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:30:58.139639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:58.139611    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:58.140059 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:58.139984    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:58.140556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:58.140538    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:30:58.539643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:58.539546    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:58.801406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:30:58.801318    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:03.358848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:03.358797    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.046573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:04.046544    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"]
Oct 30 22:31:04.095751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:04.095715    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cded56eb-qmjmq"]
Oct 30 22:31:05.020229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:05.020195    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9603c8-a68d-4528-927d-02e3c9e54f07" path="/var/lib/kubelet/pods/9f9603c8-a68d-4528-927d-02e3c9e54f07/volumes"
Oct 30 22:31:08.157360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:08.157329    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:12.998812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:12.998691    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:15.048605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:15.048572    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"]
Oct 30 22:31:15.097577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:15.097545    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8172f7e-5kbwv"]
Oct 30 22:31:16.589020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:16.588986    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:17.019026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:17.018997    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980fd268-584f-46ab-9b0c-ecbdc26963ce" path="/var/lib/kubelet/pods/980fd268-584f-46ab-9b0c-ecbdc26963ce/volumes"
Oct 30 22:31:18.202847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:18.202810    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.524525 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:31:20.524475    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:31:22.882391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:22.882344    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.568939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:24.568897    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.852460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:24.852386    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.011924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:25.011889    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.099498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:25.099464    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"]
Oct 30 22:31:25.149290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:25.149261    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b563fd-gd27l"]
Oct 30 22:31:25.516577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:25.516492    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:27.019524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:27.019493    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189f79fd-272c-4c40-bea8-a308ebb96f29" path="/var/lib/kubelet/pods/189f79fd-272c-4c40-bea8-a308ebb96f29/volumes"
Oct 30 22:31:28.886428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:28.886394    4410 scope.go:122] "RemoveContainer" containerID="423bcc3a2bcef9c8fe2b65d0fdb3e1540d7c0150374be7b3f1c53cf43c5ebeba"
Oct 30 22:31:28.895759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:28.895739    4410 scope.go:122] "RemoveContainer" containerID="2a5c3ce92d3f20306dc3630453460775258b6ac932bfa2d0c0d5b9f6a2d1fa84"
Oct 30 22:31:28.904450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:28.904427    4410 scope.go:122] "RemoveContainer" containerID="176c18d90fbb8aaf4a64485c5eb8c7f82cef83c974c4219698984c74a5716442"
Oct 30 22:31:28.916061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:28.916038    4410 scope.go:122] "RemoveContainer" containerID="98898a27fd35922db1d477eceaa8d86d120c9f0e14b0d2310604f65b0530338d"
Oct 30 22:31:29.993883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:29.993841    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.206376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:30.206340    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.357675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:30.357559    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:31.046436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:31.046407    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"]
Oct 30 22:31:31.095869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:31.095839    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1ea7d3-85fpg"]
Oct 30 22:31:31.451445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:31.451409    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:31.647320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:31.647286    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:31.806124 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:31.806026    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.019640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:33.019609    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f5a9d0-fa76-4fff-801f-ba633ef4e7e0" path="/var/lib/kubelet/pods/84f5a9d0-fa76-4fff-801f-ba633ef4e7e0/volumes"
Oct 30 22:31:34.899085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:34.899045    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.067179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:35.067139    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.297900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:35.297821    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.461269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:35.461233    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:36.332962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:36.332926    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:37.040594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.040563    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"]
Oct 30 22:31:37.045506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.045486    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.136552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.136519    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmtg\" (UniqueName: \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.136702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.136583    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.136797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.136752    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.146477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.146447    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:37.146711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.146519    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:37.146844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.146565    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:37.146844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.146617    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:31:37.151305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.151281    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"]
Oct 30 22:31:37.237687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.237657    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.237893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.237805    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmtg\" (UniqueName: \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.237893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.237863    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.238386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.238363    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.240281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.240259    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.294728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.294671    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmtg\" (UniqueName: \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\") pod \"maas-api-revoke-sub-75b912aa-xxs9f\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.358443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.358417    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:37.739117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:37.739083    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"]
Oct 30 22:31:37.739356 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:31:37.739331    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cd5595_07e4_4e4f_b9c0_3510c02f4183.slice/crio-f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa WatchSource:0}: Error finding container f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa: Status 404 returned error can't find the container with id f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa
Oct 30 22:31:38.042128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.042046    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f" event={"ID":"a5cd5595-07e4-4e4f-b9c0-3510c02f4183","Type":"ContainerStarted","Data":"f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa"}
Oct 30 22:31:38.220796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.220748    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"]
Oct 30 22:31:38.224372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.224350    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.281887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.281858    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"]
Oct 30 22:31:38.353804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.353710    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.353804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.353759    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.353977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.353929    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24n8\" (UniqueName: \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.454799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.454754    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24n8\" (UniqueName: \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.455224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.454857    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.455224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.454902    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.455605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.455585    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.457214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.457191    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.511298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.511270    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24n8\" (UniqueName: \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\") pod \"maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.538364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.538339    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:38.921361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:38.921336    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"]
Oct 30 22:31:38.922984 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:31:38.922958    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97682c0d_302b_4528_895d_76ceb652509e.slice/crio-7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d WatchSource:0}: Error finding container 7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d: Status 404 returned error can't find the container with id 7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d
Oct 30 22:31:39.049886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:39.049846    4410 generic.go:356] "Generic (PLEG): container finished" podID="a5cd5595-07e4-4e4f-b9c0-3510c02f4183" containerID="10d6f8d70eeb8e5af5023da1b18478c456403ed024e2eb1b9a730f5bbe64a9ba" exitCode=0
Oct 30 22:31:39.050053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:39.049940    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f" event={"ID":"a5cd5595-07e4-4e4f-b9c0-3510c02f4183","Type":"ContainerDied","Data":"10d6f8d70eeb8e5af5023da1b18478c456403ed024e2eb1b9a730f5bbe64a9ba"}
Oct 30 22:31:39.051336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:39.051312    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl" event={"ID":"97682c0d-302b-4528-895d-76ceb652509e","Type":"ContainerStarted","Data":"7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d"}
Oct 30 22:31:40.044296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.044268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:40.059623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.059592    4410 generic.go:356] "Generic (PLEG): container finished" podID="97682c0d-302b-4528-895d-76ceb652509e" containerID="1124127ca4fd50d9d2c92feb3c15ac4b66a5f7cb4e95a431dfcba41c5453c464" exitCode=0
Oct 30 22:31:40.059763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.059687    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl" event={"ID":"97682c0d-302b-4528-895d-76ceb652509e","Type":"ContainerDied","Data":"1124127ca4fd50d9d2c92feb3c15ac4b66a5f7cb4e95a431dfcba41c5453c464"}
Oct 30 22:31:40.202754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.202691    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:40.272725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.272694    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\") pod \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") "
Oct 30 22:31:40.272858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.272748    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\" (UniqueName: \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\") pod \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") "
Oct 30 22:31:40.272858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.272809    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\") pod \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\" (UID: \"a5cd5595-07e4-4e4f-b9c0-3510c02f4183\") "
Oct 30 22:31:40.273079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.273058    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca" pod "a5cd5595-07e4-4e4f-b9c0-3510c02f4183" (UID: "a5cd5595-07e4-4e4f-b9c0-3510c02f4183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:40.273213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.273199    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:40.274923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.274899    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca" pod "a5cd5595-07e4-4e4f-b9c0-3510c02f4183" (UID: "a5cd5595-07e4-4e4f-b9c0-3510c02f4183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:40.275026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.274924    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg" pod "a5cd5595-07e4-4e4f-b9c0-3510c02f4183" (UID: "a5cd5595-07e4-4e4f-b9c0-3510c02f4183"). InnerVolumeSpecName "kube-api-access-4hmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:40.373948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.373919    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmtg\" (UniqueName: \"kubernetes.io/projected/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-kube-api-access-4hmtg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:40.373948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:40.373944    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd5595-07e4-4e4f-b9c0-3510c02f4183-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.066982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.066956    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"
Oct 30 22:31:41.067397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.066995    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f" event={"ID":"a5cd5595-07e4-4e4f-b9c0-3510c02f4183","Type":"ContainerDied","Data":"f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa"}
Oct 30 22:31:41.067397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.067045    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5eb9f1d602cb9d0849dd32685ffe022c4ac93d1db7e1fc3b4de1e98204e84fa"
Oct 30 22:31:41.212628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.212606    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:41.283178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.283115    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\") pod \"97682c0d-302b-4528-895d-76ceb652509e\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") "
Oct 30 22:31:41.283178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.283173    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\") pod \"97682c0d-302b-4528-895d-76ceb652509e\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") "
Oct 30 22:31:41.283337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.283268    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\" (UniqueName: \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\") pod \"97682c0d-302b-4528-895d-76ceb652509e\" (UID: \"97682c0d-302b-4528-895d-76ceb652509e\") "
Oct 30 22:31:41.283518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.283490    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca" pod "97682c0d-302b-4528-895d-76ceb652509e" (UID: "97682c0d-302b-4528-895d-76ceb652509e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:41.285195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.285173    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8" pod "97682c0d-302b-4528-895d-76ceb652509e" (UID: "97682c0d-302b-4528-895d-76ceb652509e"). InnerVolumeSpecName "kube-api-access-r24n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:41.285285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.285170    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca" pod "97682c0d-302b-4528-895d-76ceb652509e" (UID: "97682c0d-302b-4528-895d-76ceb652509e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:41.384656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.384615    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97682c0d-302b-4528-895d-76ceb652509e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.384656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.384655    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97682c0d-302b-4528-895d-76ceb652509e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.384656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:41.384666    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24n8\" (UniqueName: \"kubernetes.io/projected/97682c0d-302b-4528-895d-76ceb652509e-kube-api-access-r24n8\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:42.075395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.075355    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl" event={"ID":"97682c0d-302b-4528-895d-76ceb652509e","Type":"ContainerDied","Data":"7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d"}
Oct 30 22:31:42.075395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.075397    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"
Oct 30 22:31:42.075885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.075414    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e197b09a3c95ef6662c7d79ea9c1b3aa89229bf53ef97c23c185872c781ef4d"
Oct 30 22:31:42.289514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.289481    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:42.505598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.505558    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:42.559228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.559191    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:31:42.559489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.559449    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" containerID="cri-o://89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db" gracePeriod=30
Oct 30 22:31:42.670713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.670563    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:42.732478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.732443    4410 patch_prober.go:29] interesting pod/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:42.732630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:42.732495    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:43.431624 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.431594    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"]
Oct 30 22:31:43.432049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432035    4410 cpu_manager.go:414] "containerMap: removing container" podUID="a5cd5595-07e4-4e4f-b9c0-3510c02f4183" containerName="revoke-keys"
Oct 30 22:31:43.432115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432050    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cd5595-07e4-4e4f-b9c0-3510c02f4183" containerName="revoke-keys"
Oct 30 22:31:43.432115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432066    4410 cpu_manager.go:414] "containerMap: removing container" podUID="97682c0d-302b-4528-895d-76ceb652509e" containerName="revoke-keys"
Oct 30 22:31:43.432115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432071    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97682c0d-302b-4528-895d-76ceb652509e" containerName="revoke-keys"
Oct 30 22:31:43.432219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432170    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cd5595-07e4-4e4f-b9c0-3510c02f4183" containerName="revoke-keys"
Oct 30 22:31:43.432219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.432178    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="97682c0d-302b-4528-895d-76ceb652509e" containerName="revoke-keys"
Oct 30 22:31:43.435596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.435580    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.502131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.502095    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"]
Oct 30 22:31:43.510272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.510246    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.510399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.510313    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kgc\" (UniqueName: \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.510450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.510417    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.543735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.543701    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:43.543874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.543841    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:31:43.543932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.543915    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:43.543977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.543925    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:43.611660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.611639    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.611798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.611697    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kgc\" (UniqueName: \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.611798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.611721    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.612383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.612359    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.613960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.613942    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.668870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.668843    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kgc\" (UniqueName: \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\") pod \"maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:43.747924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:43.747854    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:44.120108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:44.120079    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"]
Oct 30 22:31:44.121442 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:31:44.121418    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a56efef_da55_4a04_b1bb_c6d99d28d771.slice/crio-b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b WatchSource:0}: Error finding container b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b: Status 404 returned error can't find the container with id b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b
Oct 30 22:31:45.098271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:45.098234    4410 generic.go:356] "Generic (PLEG): container finished" podID="9a56efef-da55-4a04-b1bb-c6d99d28d771" containerID="e9ce67e74a0dacc38b3b08512b5a5cbf9616d6cffd07483ec3d33141af1221df" exitCode=0
Oct 30 22:31:45.098691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:45.098318    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk" event={"ID":"9a56efef-da55-4a04-b1bb-c6d99d28d771","Type":"ContainerDied","Data":"e9ce67e74a0dacc38b3b08512b5a5cbf9616d6cffd07483ec3d33141af1221df"}
Oct 30 22:31:45.098691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:45.098371    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk" event={"ID":"9a56efef-da55-4a04-b1bb-c6d99d28d771","Type":"ContainerStarted","Data":"b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b"}
Oct 30 22:31:45.339350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:45.339265    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:46.251479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.251456    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:46.340804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.340765    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\" (UniqueName: \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\") pod \"9a56efef-da55-4a04-b1bb-c6d99d28d771\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") "
Oct 30 22:31:46.340924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.340825    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\") pod \"9a56efef-da55-4a04-b1bb-c6d99d28d771\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") "
Oct 30 22:31:46.340924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.340854    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\") pod \"9a56efef-da55-4a04-b1bb-c6d99d28d771\" (UID: \"9a56efef-da55-4a04-b1bb-c6d99d28d771\") "
Oct 30 22:31:46.341170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.341147    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca" pod "9a56efef-da55-4a04-b1bb-c6d99d28d771" (UID: "9a56efef-da55-4a04-b1bb-c6d99d28d771"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:46.342792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.342756    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc" pod "9a56efef-da55-4a04-b1bb-c6d99d28d771" (UID: "9a56efef-da55-4a04-b1bb-c6d99d28d771"). InnerVolumeSpecName "kube-api-access-26kgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:46.342893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.342816    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca" pod "9a56efef-da55-4a04-b1bb-c6d99d28d771" (UID: "9a56efef-da55-4a04-b1bb-c6d99d28d771"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:46.442094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.442063    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kgc\" (UniqueName: \"kubernetes.io/projected/9a56efef-da55-4a04-b1bb-c6d99d28d771-kube-api-access-26kgc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:46.442094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.442092    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:46.442274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:46.442102    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a56efef-da55-4a04-b1bb-c6d99d28d771-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.111676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.111643    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"
Oct 30 22:31:47.111881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.111691    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk" event={"ID":"9a56efef-da55-4a04-b1bb-c6d99d28d771","Type":"ContainerDied","Data":"b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b"}
Oct 30 22:31:47.111881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.111741    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ad152aa940c84884b5fbea8dc4d30fff4d153c55113b75f5c4b041d75c2b2b"
Oct 30 22:31:47.820630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.820609    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:31:47.957664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957634    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.957836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957686    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.957836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957711    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.957836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957761    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957852    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957879    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957935    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957931    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958034    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.957992    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958137    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958148    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\") pod \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\" (UID: \"d895c5f9-183f-4a98-bdf2-49aab76ea9af\") "
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958156    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:47.958215 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958176    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:47.958652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958617    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.958652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958643    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.958828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958659    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.958828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958675    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.958828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.958689    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.960119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.960092    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:47.960119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.960088    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:47.960265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.960118    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "kube-api-access-7zmjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:47.960265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:47.960163    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo" pod "d895c5f9-183f-4a98-bdf2-49aab76ea9af" (UID: "d895c5f9-183f-4a98-bdf2-49aab76ea9af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:48.059936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.059902    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:48.059936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.059932    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:48.059936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.059942    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:48.060160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.059951    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmjl\" (UniqueName: \"kubernetes.io/projected/d895c5f9-183f-4a98-bdf2-49aab76ea9af-kube-api-access-7zmjl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:48.119583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.119553    4410 generic.go:356] "Generic (PLEG): container finished" podID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerID="89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db" exitCode=0
Oct 30 22:31:48.119750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.119618    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"
Oct 30 22:31:48.119750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.119635    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerDied","Data":"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"}
Oct 30 22:31:48.119750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.119688    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l" event={"ID":"d895c5f9-183f-4a98-bdf2-49aab76ea9af","Type":"ContainerDied","Data":"689d3ccb4ab39a6e9481eb4fc737a82401db31174643f0b3db6e01d37ead5506"}
Oct 30 22:31:48.119750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.119710    4410 scope.go:122] "RemoveContainer" containerID="89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"
Oct 30 22:31:48.130559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.130543    4410 scope.go:122] "RemoveContainer" containerID="051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"
Oct 30 22:31:48.139585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.139566    4410 scope.go:122] "RemoveContainer" containerID="89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"
Oct 30 22:31:48.139850 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:31:48.139830    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db\": container with ID starting with 89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db not found: ID does not exist" containerID="89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"
Oct 30 22:31:48.139919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.139857    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db"} err="failed to get container status \"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db\": rpc error: code = NotFound desc = could not find container \"89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db\": container with ID starting with 89fd9046658dc4f0e2b2dba26c7af34942baf0810f26aac44a33ee5d986091db not found: ID does not exist"
Oct 30 22:31:48.139919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.139874    4410 scope.go:122] "RemoveContainer" containerID="051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"
Oct 30 22:31:48.140086 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:31:48.140070    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5\": container with ID starting with 051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5 not found: ID does not exist" containerID="051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"
Oct 30 22:31:48.140156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.140091    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5"} err="failed to get container status \"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5\": rpc error: code = NotFound desc = could not find container \"051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5\": container with ID starting with 051c03288b5bee92f8ab031bf4f34c9ac91f1cfcea4fdf3cc4fa5a3722b667e5 not found: ID does not exist"
Oct 30 22:31:48.288686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.288597    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:31:48.292787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:48.292756    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-086575-openshift-default-6cc587cbd9-nhg8l"]
Oct 30 22:31:49.019063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:49.019022    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" path="/var/lib/kubelet/pods/d895c5f9-183f-4a98-bdf2-49aab76ea9af/volumes"
Oct 30 22:31:49.242285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:49.242255    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:50.068099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:50.068066    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:50.522639 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:31:50.522593    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:31:50.637181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:50.637152    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:52.048085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:52.048054    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"]
Oct 30 22:31:52.097241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:52.097208    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4389fcd9-5gghr"]
Oct 30 22:31:52.562726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:52.562684    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:52.723537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:52.723507    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:53.019748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:53.019714    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e08019-b2dc-4e45-a403-9f5c115e6013" path="/var/lib/kubelet/pods/07e08019-b2dc-4e45-a403-9f5c115e6013/volumes"
Oct 30 22:31:54.855502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:54.855466    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:55.061483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:55.061447    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:55.212918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:55.212885    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.418686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:56.418653    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.632765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:56.632727    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.778733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:56.778640    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.904722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:56.904689    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"]
Oct 30 22:31:56.954873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:56.954842    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-42e8a1-fkptl"]
Oct 30 22:31:57.020399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:57.020365    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97682c0d-302b-4528-895d-76ceb652509e" path="/var/lib/kubelet/pods/97682c0d-302b-4528-895d-76ceb652509e/volumes"
Oct 30 22:31:57.302331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:57.302297    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.894585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:31:59.894555    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.364669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:00.364581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:02.045161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:02.045124    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.224682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:03.224645    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.444433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:03.444397    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.616353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:03.616269    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.376199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376166    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"]
Oct 30 22:32:04.376589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376580    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:04.376633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376591    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:04.376633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376626    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9a56efef-da55-4a04-b1bb-c6d99d28d771" containerName="revoke-keys"
Oct 30 22:32:04.376633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376631    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a56efef-da55-4a04-b1bb-c6d99d28d771" containerName="revoke-keys"
Oct 30 22:32:04.376740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376695    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:04.376740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376705    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:04.376740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.376715    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a56efef-da55-4a04-b1bb-c6d99d28d771" containerName="revoke-keys"
Oct 30 22:32:04.380212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.380189    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.483123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.483087    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:04.483123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.483091    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:32:04.483399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.483091    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:04.483399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.483253    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:04.486533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.486511    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"]
Oct 30 22:32:04.537557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.537527    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.537702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.537562    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.537805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.537758    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ccg\" (UniqueName: \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.638672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.638648    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.638821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.638681    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.638821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.638750    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ccg\" (UniqueName: \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.639394 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.639375    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.641105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.641084    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.695078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.695047    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ccg\" (UniqueName: \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\") pod \"maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:04.993276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:04.993191    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:05.366490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:05.366457    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"]
Oct 30 22:32:05.367748 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:32:05.367721    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc155bd33_4933_4c1a_941b_b1dab215fe93.slice/crio-43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f WatchSource:0}: Error finding container 43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f: Status 404 returned error can't find the container with id 43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f
Oct 30 22:32:06.235385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:06.235351    4410 generic.go:356] "Generic (PLEG): container finished" podID="c155bd33-4933-4c1a-941b-b1dab215fe93" containerID="0502b216b2d4803cc04fb11cf3c557a76a850cf02508389ba3631369bebbdd60" exitCode=0
Oct 30 22:32:06.235688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:06.235427    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2" event={"ID":"c155bd33-4933-4c1a-941b-b1dab215fe93","Type":"ContainerDied","Data":"0502b216b2d4803cc04fb11cf3c557a76a850cf02508389ba3631369bebbdd60"}
Oct 30 22:32:06.235688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:06.235480    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2" event={"ID":"c155bd33-4933-4c1a-941b-b1dab215fe93","Type":"ContainerStarted","Data":"43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f"}
Oct 30 22:32:07.341063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.341016    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.389181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.389160    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:07.574399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.574315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\" (UniqueName: \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\") pod \"c155bd33-4933-4c1a-941b-b1dab215fe93\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") "
Oct 30 22:32:07.574399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.574358    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\") pod \"c155bd33-4933-4c1a-941b-b1dab215fe93\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") "
Oct 30 22:32:07.574399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.574391    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\") pod \"c155bd33-4933-4c1a-941b-b1dab215fe93\" (UID: \"c155bd33-4933-4c1a-941b-b1dab215fe93\") "
Oct 30 22:32:07.574722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.574699    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca" pod "c155bd33-4933-4c1a-941b-b1dab215fe93" (UID: "c155bd33-4933-4c1a-941b-b1dab215fe93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:07.576505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.576484    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca" pod "c155bd33-4933-4c1a-941b-b1dab215fe93" (UID: "c155bd33-4933-4c1a-941b-b1dab215fe93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:07.576594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.576523    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg" pod "c155bd33-4933-4c1a-941b-b1dab215fe93" (UID: "c155bd33-4933-4c1a-941b-b1dab215fe93"). InnerVolumeSpecName "kube-api-access-h2ccg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:07.675765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.675738    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ccg\" (UniqueName: \"kubernetes.io/projected/c155bd33-4933-4c1a-941b-b1dab215fe93-kube-api-access-h2ccg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:07.675765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.675762    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:07.675765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:07.675785    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c155bd33-4933-4c1a-941b-b1dab215fe93-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:08.253180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:08.253135    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2" event={"ID":"c155bd33-4933-4c1a-941b-b1dab215fe93","Type":"ContainerDied","Data":"43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f"}
Oct 30 22:32:08.253180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:08.253177    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"
Oct 30 22:32:08.253473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:08.253231    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e9c43124018cbdc1be2fbfa92c296824cbbf103259c53ed63c0d7e1734cf8f"
Oct 30 22:32:09.056126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:09.056096    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"]
Oct 30 22:32:09.105057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:09.105028    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c237c82d-dqxbt"]
Oct 30 22:32:11.018932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:11.018898    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b177c16-c678-4d1e-9984-68ff3c2af978" path="/var/lib/kubelet/pods/6b177c16-c678-4d1e-9984-68ff3c2af978/volumes"
Oct 30 22:32:12.294628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:12.294594    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:12.498588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:12.498554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:12.654657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:12.654611    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:15.584223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:15.584185    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:15.783503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:15.783468    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.742282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:17.742249    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:19.067091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:19.067048    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:32:19.450661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:19.450623    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:19.659467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:19.659432    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.525705 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:32:20.525658    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:32:22.538839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:22.538799    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:24.067990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.067953    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" podUID="8b78d935-14be-43ba-91fd-6f9d8eec7248" containerName="payload-processing" containerID="cri-o://b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0" gracePeriod=25
Oct 30 22:32:24.326578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.326525    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:32:24.352022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.351990    4410 generic.go:356] "Generic (PLEG): container finished" podID="8b78d935-14be-43ba-91fd-6f9d8eec7248" containerID="b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0" exitCode=0
Oct 30 22:32:24.352161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.352042    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"
Oct 30 22:32:24.352161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.352074    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" event={"ID":"8b78d935-14be-43ba-91fd-6f9d8eec7248","Type":"ContainerDied","Data":"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"}
Oct 30 22:32:24.352161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.352124    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff" event={"ID":"8b78d935-14be-43ba-91fd-6f9d8eec7248","Type":"ContainerDied","Data":"165c5bad727bb8e1ea2e9982bd80f4b51b2040b53294e6064850325af6f7e698"}
Oct 30 22:32:24.352277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.352156    4410 scope.go:122] "RemoveContainer" containerID="b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"
Oct 30 22:32:24.361623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.361591    4410 scope.go:122] "RemoveContainer" containerID="b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"
Oct 30 22:32:24.361919 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:32:24.361899    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0\": container with ID starting with b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0 not found: ID does not exist" containerID="b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"
Oct 30 22:32:24.361976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.361928    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0"} err="failed to get container status \"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0\": rpc error: code = NotFound desc = could not find container \"b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0\": container with ID starting with b480290abba0711c494d775f46540233608445fc13cdc9dc8578dd2942acebc0 not found: ID does not exist"
Oct 30 22:32:24.461167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.461135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\") pod \"8b78d935-14be-43ba-91fd-6f9d8eec7248\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") "
Oct 30 22:32:24.461355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.461336    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\" (UniqueName: \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\") pod \"8b78d935-14be-43ba-91fd-6f9d8eec7248\" (UID: \"8b78d935-14be-43ba-91fd-6f9d8eec7248\") "
Oct 30 22:32:24.461477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.461455    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume" pod "8b78d935-14be-43ba-91fd-6f9d8eec7248" (UID: "8b78d935-14be-43ba-91fd-6f9d8eec7248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:24.461878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.461858    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b78d935-14be-43ba-91fd-6f9d8eec7248-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:24.463336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.463312    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg" pod "8b78d935-14be-43ba-91fd-6f9d8eec7248" (UID: "8b78d935-14be-43ba-91fd-6f9d8eec7248"). InnerVolumeSpecName "kube-api-access-494cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:24.562659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.562620    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494cg\" (UniqueName: \"kubernetes.io/projected/8b78d935-14be-43ba-91fd-6f9d8eec7248-kube-api-access-494cg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:24.776598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.776562    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:32:24.782459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:24.782431    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3bbd9b-69c69f6b49-z4kff"]
Oct 30 22:32:25.019929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:25.019858    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b78d935-14be-43ba-91fd-6f9d8eec7248" path="/var/lib/kubelet/pods/8b78d935-14be-43ba-91fd-6f9d8eec7248/volumes"
Oct 30 22:32:27.433252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:27.433217    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.950514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:28.950473    4410 scope.go:122] "RemoveContainer" containerID="c6eb81a2cd20dc54a394cdd718fe0bb85195c76e75e21cd8b897b4165a85d83f"
Oct 30 22:32:28.959946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:28.959923    4410 scope.go:122] "RemoveContainer" containerID="f1c126ff5533337f202ae6ac5e83ac061d29be4abea0f4b8a2af423d29a54c7f"
Oct 30 22:32:28.971889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:28.971869    4410 scope.go:122] "RemoveContainer" containerID="95b108d801eb2ab7394565041cb989dc96658716f9fda26f4982e52d8335b078"
Oct 30 22:32:28.980129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:28.980108    4410 scope.go:122] "RemoveContainer" containerID="a938843f1e3b5a18557be688945d9d8aa67061b4063d25e2fae7dce5b5b0e513"
Oct 30 22:32:28.987925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:28.987907    4410 scope.go:122] "RemoveContainer" containerID="0037bea7a5e05e8e74c555d860ba2cb3b2fc80b9997e9e0f59f324192775e9c0"
Oct 30 22:32:29.586312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:29.586280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:29.991178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:29.991144    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:30.048987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:30.048957    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"]
Oct 30 22:32:30.098331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:30.098299    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4050ee-c8fws"]
Oct 30 22:32:30.701585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:30.701549    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:30.853535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:30.853490    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:31.001558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:31.001473    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:31.021264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:31.021224    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eeda028-d249-4b4f-bd20-6a4e9ba98610" path="/var/lib/kubelet/pods/6eeda028-d249-4b4f-bd20-6a4e9ba98610/volumes"
Oct 30 22:32:32.737047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:32.737009    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:34.872241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:34.872203    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.083102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:35.083072    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.239611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:35.239568    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:37.886429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:37.886389    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.056321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.056278    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.228314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.228280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.391266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.391225    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.549510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.549424    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.798660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.798623    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:32:38.799664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.799594    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8b78d935-14be-43ba-91fd-6f9d8eec7248" containerName="payload-processing"
Oct 30 22:32:38.799858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.799841    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b78d935-14be-43ba-91fd-6f9d8eec7248" containerName="payload-processing"
Oct 30 22:32:38.799977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.799966    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:38.800066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.800056    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d895c5f9-183f-4a98-bdf2-49aab76ea9af" containerName="istio-proxy"
Oct 30 22:32:38.800174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.800165    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c155bd33-4933-4c1a-941b-b1dab215fe93" containerName="revoke-keys"
Oct 30 22:32:38.800248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.800239    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c155bd33-4933-4c1a-941b-b1dab215fe93" containerName="revoke-keys"
Oct 30 22:32:38.800465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.800453    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b78d935-14be-43ba-91fd-6f9d8eec7248" containerName="payload-processing"
Oct 30 22:32:38.800562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.800552    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c155bd33-4933-4c1a-941b-b1dab215fe93" containerName="revoke-keys"
Oct 30 22:32:38.812809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.806339    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:38.846883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.846854    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:32:38.886339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.886307    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6c788d\""
Oct 30 22:32:38.886339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.886321    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6c788d-dockercfg-fw5vz\""
Oct 30 22:32:38.917544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.917501    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthjb\" (UniqueName: \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:38.917737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:38.917716    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.019419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.019385    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.019627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.019474    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthjb\" (UniqueName: \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.020494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.020464    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.044393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.044364    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.078152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.078079    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthjb\" (UniqueName: \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\") pod \"payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.154321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.154295    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:39.242677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.242570    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.531705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:39.531677    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:32:39.533633 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:32:39.533606    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eba5e00_f84a_4644_ab13_601b2c57fffd.slice/crio-5e12f17bc675287e42d19e0a61e1a0f6caada51fbdace0f2fdb1d84421aed7b9 WatchSource:0}: Error finding container 5e12f17bc675287e42d19e0a61e1a0f6caada51fbdace0f2fdb1d84421aed7b9: Status 404 returned error can't find the container with id 5e12f17bc675287e42d19e0a61e1a0f6caada51fbdace0f2fdb1d84421aed7b9
Oct 30 22:32:40.460304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:40.460271    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" event={"ID":"2eba5e00-f84a-4644-ab13-601b2c57fffd","Type":"ContainerStarted","Data":"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"}
Oct 30 22:32:40.460760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:40.460321    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" event={"ID":"2eba5e00-f84a-4644-ab13-601b2c57fffd","Type":"ContainerStarted","Data":"5e12f17bc675287e42d19e0a61e1a0f6caada51fbdace0f2fdb1d84421aed7b9"}
Oct 30 22:32:40.460760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:40.460405    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:40.575604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:40.575549    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" podStartSLOduration=2.575536374 podStartE2EDuration="2.575536374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:40.572585022 +0000 UTC m=+3013.661322827" watchObservedRunningTime="2026-10-30 22:32:40.575536374 +0000 UTC m=+3013.664274191"
Oct 30 22:32:42.649420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:42.649385    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:42.929542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:42.929440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.108035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:43.107999    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.266439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:43.266353    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:44.317639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:44.317602    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:44.651103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:44.651070    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:44.800704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:44.800665    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:44.850281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:44.850251    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"]
Oct 30 22:32:44.852955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:44.852929    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-37a390-2qznk"]
Oct 30 22:32:45.018786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:45.018698    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a56efef-da55-4a04-b1bb-c6d99d28d771" path="/var/lib/kubelet/pods/9a56efef-da55-4a04-b1bb-c6d99d28d771/volumes"
Oct 30 22:32:45.275181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:45.275100    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:46.861814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:46.861755    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:47.074234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:47.074205    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:49.242886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:49.242852    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:50.521270 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:32:50.521236    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:32:50.819466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:50.819385    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.040622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.040587    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.046380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.046347    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"]
Oct 30 22:32:51.095453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.095387    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30631cf6-r8p6n"]
Oct 30 22:32:51.189750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.189698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.306309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.306228    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"]
Oct 30 22:32:51.310721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.310687    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3bbd9b-dchr2"]
Oct 30 22:32:51.342390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.342358    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.469767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.469741    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:32:51.620114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.620054    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.774087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:51.773986    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:53.019884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:53.019845    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c155bd33-4933-4c1a-941b-b1dab215fe93" path="/var/lib/kubelet/pods/c155bd33-4933-4c1a-941b-b1dab215fe93/volumes"
Oct 30 22:32:53.020386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:53.020366    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7084bd7-ae5d-4df9-ae6e-b51332042b92" path="/var/lib/kubelet/pods/c7084bd7-ae5d-4df9-ae6e-b51332042b92/volumes"
Oct 30 22:32:56.109261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.109230    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:56.286294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.286255    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:56.435154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.435119    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:56.569245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.569211    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"]
Oct 30 22:32:56.574396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.574376    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.628829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.628799    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"]
Oct 30 22:32:56.674885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.674860    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:56.675012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.674883    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:56.675012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.674920    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:32:56.675716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.675702    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:56.719367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.719309    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhkh\" (UniqueName: \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.719367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.719359    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.719518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.719433    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.820942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.820908    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhkh\" (UniqueName: \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.821099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.820968    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.821099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.821011    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.821647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.821622    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.823323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.823305    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.877306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.877280    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhkh\" (UniqueName: \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\") pod \"maas-api-revoke-sub-7ce0314d-52tmh\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:56.886987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:56.886964    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:57.259839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.259815    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"]
Oct 30 22:32:57.261011 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:32:57.260984    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cd8993_d4f9_4f51_80c6_f3ceedd77be7.slice/crio-4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba WatchSource:0}: Error finding container 4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba: Status 404 returned error can't find the container with id 4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba
Oct 30 22:32:57.578037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.578006    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh" event={"ID":"95cd8993-d4f9-4f51-80c6-f3ceedd77be7","Type":"ContainerStarted","Data":"49c1a4620f3b634092cf841a0503b3a3e6c2c54049c857f01646b8948e74faef"}
Oct 30 22:32:57.578183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.578055    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh" event={"ID":"95cd8993-d4f9-4f51-80c6-f3ceedd77be7","Type":"ContainerStarted","Data":"4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba"}
Oct 30 22:32:57.694096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.694043    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh" podStartSLOduration=1.694029699 podStartE2EDuration="1.694029699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:57.691063652 +0000 UTC m=+3030.779801456" watchObservedRunningTime="2026-10-30 22:32:57.694029699 +0000 UTC m=+3030.782767533"
Oct 30 22:32:57.753332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.753302    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:57.919831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:57.919767    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:58.585058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:58.585026    4410 generic.go:356] "Generic (PLEG): container finished" podID="95cd8993-d4f9-4f51-80c6-f3ceedd77be7" containerID="49c1a4620f3b634092cf841a0503b3a3e6c2c54049c857f01646b8948e74faef" exitCode=0
Oct 30 22:32:58.585466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:58.585109    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh" event={"ID":"95cd8993-d4f9-4f51-80c6-f3ceedd77be7","Type":"ContainerDied","Data":"49c1a4620f3b634092cf841a0503b3a3e6c2c54049c857f01646b8948e74faef"}
Oct 30 22:32:59.737544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.737516    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:32:59.854020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.853932    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\" (UniqueName: \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\") pod \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") "
Oct 30 22:32:59.854184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.854025    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\") pod \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") "
Oct 30 22:32:59.854251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.854183    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\") pod \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\" (UID: \"95cd8993-d4f9-4f51-80c6-f3ceedd77be7\") "
Oct 30 22:32:59.854696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.854660    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca" pod "95cd8993-d4f9-4f51-80c6-f3ceedd77be7" (UID: "95cd8993-d4f9-4f51-80c6-f3ceedd77be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:59.856103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.856073    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh" pod "95cd8993-d4f9-4f51-80c6-f3ceedd77be7" (UID: "95cd8993-d4f9-4f51-80c6-f3ceedd77be7"). InnerVolumeSpecName "kube-api-access-4nhkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:59.856248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.856216    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca" pod "95cd8993-d4f9-4f51-80c6-f3ceedd77be7" (UID: "95cd8993-d4f9-4f51-80c6-f3ceedd77be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:59.955933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.955897    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:59.955933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.955929    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhkh\" (UniqueName: \"kubernetes.io/projected/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-kube-api-access-4nhkh\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:59.956117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:32:59.955944    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cd8993-d4f9-4f51-80c6-f3ceedd77be7-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.599296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:00.599269    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"
Oct 30 22:33:00.599473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:00.599259    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh" event={"ID":"95cd8993-d4f9-4f51-80c6-f3ceedd77be7","Type":"ContainerDied","Data":"4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba"}
Oct 30 22:33:00.599473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:00.599407    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ce77efb204362e15824132168de9fa5d9ac2a78c534c8ed44e177e66208f9ba"
Oct 30 22:33:01.752669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:01.752635    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:06.507072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:06.507017    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:06.666549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:06.666515    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:08.082460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:08.082417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:11.416078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:11.416036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:16.178994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:16.178960    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:18.500794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:18.500733    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.093097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:19.093064    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.360892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:19.360795    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:20.522432 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:33:20.522394    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:33:21.015106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:21.015068    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:22.872449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:22.872408    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:23.032488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:23.032451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:23.204840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:23.204810    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.146022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.145971    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.306570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.306534    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.462239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.462134    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.702473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.702441    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:33:26.702951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.702934    4410 cpu_manager.go:414] "containerMap: removing container" podUID="95cd8993-d4f9-4f51-80c6-f3ceedd77be7" containerName="revoke-keys"
Oct 30 22:33:26.703019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.702953    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cd8993-d4f9-4f51-80c6-f3ceedd77be7" containerName="revoke-keys"
Oct 30 22:33:26.703062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.703043    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cd8993-d4f9-4f51-80c6-f3ceedd77be7" containerName="revoke-keys"
Oct 30 22:33:26.706526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.706507    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:26.810682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.810651    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-89cf28\""
Oct 30 22:33:26.810881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.810856    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-89cf28-dockercfg-5g2p6\""
Oct 30 22:33:26.813656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.813635    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:33:26.859342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.859309    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbzz\" (UniqueName: \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:26.859476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.859355    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:26.960269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.960227    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbzz\" (UniqueName: \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:26.960470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.960282    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:26.961096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:26.961068    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:27.021571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:27.021502    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbzz\" (UniqueName: \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\") pod \"payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:27.371385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:27.371298    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-89cf28-dockercfg-5g2p6\""
Oct 30 22:33:27.380439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:27.380413    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:27.752268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:27.752233    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:33:27.753366 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:33:27.753339    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e122a52_d9af_417e_92d5_83ac2c2155be.slice/crio-de2e4644662891120f1b9b8a72666654b8c6fa76d22685b2064cc42f8196b2c4 WatchSource:0}: Error finding container de2e4644662891120f1b9b8a72666654b8c6fa76d22685b2064cc42f8196b2c4: Status 404 returned error can't find the container with id de2e4644662891120f1b9b8a72666654b8c6fa76d22685b2064cc42f8196b2c4
Oct 30 22:33:27.765337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:27.765308    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" event={"ID":"3e122a52-d9af-417e-92d5-83ac2c2155be","Type":"ContainerStarted","Data":"de2e4644662891120f1b9b8a72666654b8c6fa76d22685b2064cc42f8196b2c4"}
Oct 30 22:33:28.046293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:28.046213    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"]
Oct 30 22:33:28.095626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:28.095598    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44728e89-gc6z9"]
Oct 30 22:33:28.773710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:28.773670    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" event={"ID":"3e122a52-d9af-417e-92d5-83ac2c2155be","Type":"ContainerStarted","Data":"16ed0ab553c68e09dbf6f40e2ce7b88cd435af0476721419600f6a4ef9821309"}
Oct 30 22:33:28.774149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:28.773838    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:28.890757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:28.890701    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" podStartSLOduration=2.8906887279999998 podStartE2EDuration="2.890688728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:28.888763939 +0000 UTC m=+3061.977501754" watchObservedRunningTime="2026-10-30 22:33:28.890688728 +0000 UTC m=+3061.979426559"
Oct 30 22:33:29.019608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:29.019571    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745987a6-cba7-49c4-95b6-bac1775035c0" path="/var/lib/kubelet/pods/745987a6-cba7-49c4-95b6-bac1775035c0/volumes"
Oct 30 22:33:29.039584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:29.039510    4410 scope.go:122] "RemoveContainer" containerID="e1bf938985439c5d358694b24924f0ba4c0910db6e32df5e7ac6889dfc72ccb2"
Oct 30 22:33:29.049034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:29.049015    4410 scope.go:122] "RemoveContainer" containerID="baa0acac1f38bf4efd277a05b8ef415c8a14a1ad57b803c938dd0a291cc37e90"
Oct 30 22:33:29.057469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:29.057439    4410 scope.go:122] "RemoveContainer" containerID="0ad601ddc8b8463109a28c871c500ac88a459ae684015c9011e0d82b7ab15126"
Oct 30 22:33:31.047000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:31.046969    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:31.622452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:31.622420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:36.768524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:36.768495    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:36.947450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:36.947420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:37.239216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:37.239181    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:38.905418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.905375    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:33:38.909133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.909107    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.965744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.965707    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\""
Oct 30 22:33:38.993670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.993637    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.993851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.993689    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.993851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.993726    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9pk\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994020    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994063    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994178    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994272    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:38.994363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:38.994344    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.029083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.029048    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:33:39.095795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095745    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.095795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095794    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095820    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095845    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095868    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095910    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095952    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.095993    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096050    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9pk\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096216    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096267    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096288    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096553    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.096695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.096672    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.098123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.098099    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.098367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.098350    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.152263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.152231    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.152441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.152425    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9pk\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.222446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.222399    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:39.651759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.651731    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:33:39.652763 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:33:39.652741    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8cd2ed_2438_4b43_a61d_d0cc5531c754.slice/crio-7c19ddfd37a0f7dd400ecd0027753dacf25ea98751d50ab12243c41b420081a2 WatchSource:0}: Error finding container 7c19ddfd37a0f7dd400ecd0027753dacf25ea98751d50ab12243c41b420081a2: Status 404 returned error can't find the container with id 7c19ddfd37a0f7dd400ecd0027753dacf25ea98751d50ab12243c41b420081a2
Oct 30 22:33:39.655341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.655305    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:33:39.655431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.655370    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:33:39.655431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.655396    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:33:39.782926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.782897    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:33:39.853155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.853116    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" event={"ID":"fd8cd2ed-2438-4b43-a61d-d0cc5531c754","Type":"ContainerStarted","Data":"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"}
Oct 30 22:33:39.853305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:39.853217    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" event={"ID":"fd8cd2ed-2438-4b43-a61d-d0cc5531c754","Type":"ContainerStarted","Data":"7c19ddfd37a0f7dd400ecd0027753dacf25ea98751d50ab12243c41b420081a2"}
Oct 30 22:33:40.021973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:40.021861    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podStartSLOduration=2.021842235 podStartE2EDuration="2.021842235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:40.018193277 +0000 UTC m=+3073.106931085" watchObservedRunningTime="2026-10-30 22:33:40.021842235 +0000 UTC m=+3073.110580038"
Oct 30 22:33:40.223620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:40.223573    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:40.225044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:40.225013    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:40.225190 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:40.225076    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:41.223060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:41.223029    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:41.223481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:41.223080    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:41.886221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:41.886181    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:42.223694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:42.223657    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:42.224216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:42.223741    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:43.071146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:43.071109    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"]
Oct 30 22:33:43.097667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:43.097634    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcd97ba-px66m"]
Oct 30 22:33:43.223294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:43.223260    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:43.223466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:43.223314    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:44.223514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:44.223477    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:44.223968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:44.223535    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:45.019503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:45.019471    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc291032-1a5e-4a2d-a121-114d14bc8bbb" path="/var/lib/kubelet/pods/cc291032-1a5e-4a2d-a121-114d14bc8bbb/volumes"
Oct 30 22:33:45.223604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:45.223569    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:45.224004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:45.223620    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:46.223098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.223060    4410 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:33:46.223312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.223121    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:33:46.664983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.664948    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"]
Oct 30 22:33:46.670064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.670042    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.672636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.672608    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:46.726980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.726936    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"]
Oct 30 22:33:46.771918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.771885    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:46.771918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.771904    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:46.772185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.771958    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:46.772284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.771903    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:33:46.805249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.805223    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7z6\" (UniqueName: \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.805385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.805357    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.805461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.805444    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.906669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.906611    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.906912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.906687    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.906912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.906840    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7z6\" (UniqueName: \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.907540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.907507    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.909375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.909350    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.963451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.963389    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7z6\" (UniqueName: \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\") pod \"maas-api-revoke-sub-11d59290-56sd8\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:46.985436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:46.985413    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:47.228434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.228360    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:47.228687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.228662    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:47.229395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.229377    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:33:47.363010 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:33:47.362981    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731bcc6a_7232_417f_ac57_5c3df3f41986.slice/crio-c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5 WatchSource:0}: Error finding container c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5: Status 404 returned error can't find the container with id c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5
Oct 30 22:33:47.400088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.400061    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"]
Oct 30 22:33:47.918327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.918274    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8" event={"ID":"731bcc6a-7232-417f-ac57-5c3df3f41986","Type":"ContainerStarted","Data":"dfcd71ae2a1523f7650c26de0861222d76ccb4d9589ab59398553ca7a22bea09"}
Oct 30 22:33:47.918817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:47.918342    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8" event={"ID":"731bcc6a-7232-417f-ac57-5c3df3f41986","Type":"ContainerStarted","Data":"c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5"}
Oct 30 22:33:48.035907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:48.035862    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8" podStartSLOduration=2.0358485 podStartE2EDuration="2.0358485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:48.032893025 +0000 UTC m=+3081.121630839" watchObservedRunningTime="2026-10-30 22:33:48.0358485 +0000 UTC m=+3081.124586302"
Oct 30 22:33:48.927635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:48.927599    4410 generic.go:356] "Generic (PLEG): container finished" podID="731bcc6a-7232-417f-ac57-5c3df3f41986" containerID="dfcd71ae2a1523f7650c26de0861222d76ccb4d9589ab59398553ca7a22bea09" exitCode=0
Oct 30 22:33:48.928098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:48.927682    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8" event={"ID":"731bcc6a-7232-417f-ac57-5c3df3f41986","Type":"ContainerDied","Data":"dfcd71ae2a1523f7650c26de0861222d76ccb4d9589ab59398553ca7a22bea09"}
Oct 30 22:33:50.073443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.073415    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:50.241385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.241319    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\") pod \"731bcc6a-7232-417f-ac57-5c3df3f41986\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") "
Oct 30 22:33:50.241525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.241399    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\" (UniqueName: \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\") pod \"731bcc6a-7232-417f-ac57-5c3df3f41986\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") "
Oct 30 22:33:50.241525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.241497    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\") pod \"731bcc6a-7232-417f-ac57-5c3df3f41986\" (UID: \"731bcc6a-7232-417f-ac57-5c3df3f41986\") "
Oct 30 22:33:50.241717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.241691    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca" pod "731bcc6a-7232-417f-ac57-5c3df3f41986" (UID: "731bcc6a-7232-417f-ac57-5c3df3f41986"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:50.241988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.241970    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.243366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.243339    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca" pod "731bcc6a-7232-417f-ac57-5c3df3f41986" (UID: "731bcc6a-7232-417f-ac57-5c3df3f41986"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:50.243462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.243429    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6" pod "731bcc6a-7232-417f-ac57-5c3df3f41986" (UID: "731bcc6a-7232-417f-ac57-5c3df3f41986"). InnerVolumeSpecName "kube-api-access-kf7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:50.343006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.342982    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7z6\" (UniqueName: \"kubernetes.io/projected/731bcc6a-7232-417f-ac57-5c3df3f41986-kube-api-access-kf7z6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.343006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.343006    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731bcc6a-7232-417f-ac57-5c3df3f41986-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.521663 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:33:50.521593    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:33:50.941147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.941118    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"
Oct 30 22:33:50.941341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.941116    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8" event={"ID":"731bcc6a-7232-417f-ac57-5c3df3f41986","Type":"ContainerDied","Data":"c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5"}
Oct 30 22:33:50.941341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:50.941283    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e7128ff133f75818da01e4ab5db0e656eefacb2760bf90923378c07fbf0bb5"
Oct 30 22:33:51.471407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:51.471374    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:55.949299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:55.949265    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:56.131572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:56.131541    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:56.472519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:33:56.472478    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:01.642314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:01.642275    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:06.046892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:06.046845    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"]
Oct 30 22:34:06.097361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:06.097331    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9322e0e-2kt6q"]
Oct 30 22:34:06.552089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:06.552058    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:07.019584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:07.019549    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc4096a-1a52-435a-b875-e39bbc39f0ed" path="/var/lib/kubelet/pods/7bc4096a-1a52-435a-b875-e39bbc39f0ed/volumes"
Oct 30 22:34:09.047430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:09.047392    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:09.482605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:09.482563    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:10.470836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:10.470798    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:10.656751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:10.656717    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:11.442527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:11.442494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:14.344163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:14.344119    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:14.577625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:14.577583    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.563741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:16.563704    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:17.583357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:17.583312    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:17.752391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:17.752358    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:18.118474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.118435    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:34:18.119273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.119249    4410 cpu_manager.go:414] "containerMap: removing container" podUID="731bcc6a-7232-417f-ac57-5c3df3f41986" containerName="revoke-keys"
Oct 30 22:34:18.119393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.119276    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731bcc6a-7232-417f-ac57-5c3df3f41986" containerName="revoke-keys"
Oct 30 22:34:18.119458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.119419    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="731bcc6a-7232-417f-ac57-5c3df3f41986" containerName="revoke-keys"
Oct 30 22:34:18.123254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.123230    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.224513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.224479    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9f7654\""
Oct 30 22:34:18.224717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.224698    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9f7654-dockercfg-95k75\""
Oct 30 22:34:18.227557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.227537    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:34:18.239126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.239097    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.239305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.239287    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sx7\" (UniqueName: \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.340194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.340167    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.340313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.340254    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sx7\" (UniqueName: \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.340851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.340834    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.396975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.396945    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sx7\" (UniqueName: \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\") pod \"payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.438825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.438798    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:18.814901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.814869    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:34:18.816076 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:34:18.816048    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694538c8_6f7b_4081_8908_59124e60c1cc.slice/crio-21355174fc807dc4aacb10d7b56aab84f4cd3771123ce4325281fe1b404b03a0 WatchSource:0}: Error finding container 21355174fc807dc4aacb10d7b56aab84f4cd3771123ce4325281fe1b404b03a0: Status 404 returned error can't find the container with id 21355174fc807dc4aacb10d7b56aab84f4cd3771123ce4325281fe1b404b03a0
Oct 30 22:34:18.926726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:18.926698    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:19.046444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.046418    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"]
Oct 30 22:34:19.049114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.049079    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ecc667-58tn4"]
Oct 30 22:34:19.128465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.128375    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" event={"ID":"694538c8-6f7b-4081-8908-59124e60c1cc","Type":"ContainerStarted","Data":"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"}
Oct 30 22:34:19.128465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.128436    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" event={"ID":"694538c8-6f7b-4081-8908-59124e60c1cc","Type":"ContainerStarted","Data":"21355174fc807dc4aacb10d7b56aab84f4cd3771123ce4325281fe1b404b03a0"}
Oct 30 22:34:19.128662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.128576    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:19.243530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:19.243470    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" podStartSLOduration=1.243455077 podStartE2EDuration="1.243455077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:19.241469672 +0000 UTC m=+3112.330207488" watchObservedRunningTime="2026-10-30 22:34:19.243455077 +0000 UTC m=+3112.332192882"
Oct 30 22:34:20.522650 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:34:20.522601    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:34:21.018916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:21.018887    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2043cadd-50ab-4024-9d6d-5a81deb32668" path="/var/lib/kubelet/pods/2043cadd-50ab-4024-9d6d-5a81deb32668/volumes"
Oct 30 22:34:21.487447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:21.487394    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:21.657352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:21.657318    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:21.903505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:21.903469    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:26.477020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:26.476983    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:29.046646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.046613    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"]
Oct 30 22:34:29.090483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.090452    4410 scope.go:122] "RemoveContainer" containerID="c8d29a9912b28de6e49a5b870052a0399ccf2339c9bb598a0b5e47580db5654a"
Oct 30 22:34:29.096313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.096282    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781d01ed-h5jdx"]
Oct 30 22:34:29.100979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.100955    4410 scope.go:122] "RemoveContainer" containerID="84c15a1d03696c4f09859abebd6445e963257cd141efd8330b9c69eea2c724a4"
Oct 30 22:34:29.109264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.109248    4410 scope.go:122] "RemoveContainer" containerID="e94b4eeb7731b01ef4a6191f043ea85e49f77f63ce005904c8e5ddbe75d2f773"
Oct 30 22:34:29.117390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.117366    4410 scope.go:122] "RemoveContainer" containerID="50849ef1cdd25433e39e00960fcc8d44485cc2b65e9fd3122ec95de003895906"
Oct 30 22:34:29.850024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:29.849941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.065860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:30.065747    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.138153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:30.138072    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:34:31.019428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:31.019397    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4a5589-fef2-43b2-b087-354036a65040" path="/var/lib/kubelet/pods/1b4a5589-fef2-43b2-b087-354036a65040/volumes"
Oct 30 22:34:31.369531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:31.369444    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:31.580558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:31.580525    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.275843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:32.275804    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:36.256603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.256575    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:34:36.260714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.260695    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347246    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347382    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347444    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347501    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzzl\" (UniqueName: \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347617 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347601    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.347666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.347654    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.365178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.365149    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a0116e\"/\"kube-root-ca.crt\""
Oct 30 22:34:36.365889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.365868    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a0116e\"/\"openshift-service-ca.crt\""
Oct 30 22:34:36.365974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.365876    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a0116e\"/\"auth-test-model-a0116e-kserve-self-signed-certs\""
Oct 30 22:34:36.365974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.365878    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a0116e\"/\"default-dockercfg-qw45t\""
Oct 30 22:34:36.368784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.368751    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:36.369288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.369269    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:34:36.451783 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.451734    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzzl\" (UniqueName: \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.451960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.451838    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.451960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.451884    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.451960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.451914    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.452089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.452006    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.452089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.452053    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.452420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.452395    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.452531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.452419    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.452591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.452526    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.454812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.454764    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.455301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.455276    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.511315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.511235    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzzl\" (UniqueName: \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\") pod \"auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") " pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.567315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.567281    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:36.573183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.573148    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:36.956708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:36.956680    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:34:36.958805 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:34:36.958763    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod258c7c37_fefd_4260_b730_cf12816855e2.slice/crio-61efafb5db693b9a39b3b2a9216315e140d1e7a2d17179afd69b3eb5f8ac547c WatchSource:0}: Error finding container 61efafb5db693b9a39b3b2a9216315e140d1e7a2d17179afd69b3eb5f8ac547c: Status 404 returned error can't find the container with id 61efafb5db693b9a39b3b2a9216315e140d1e7a2d17179afd69b3eb5f8ac547c
Oct 30 22:34:37.247756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:37.247675    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" event={"ID":"258c7c37-fefd-4260-b730-cf12816855e2","Type":"ContainerStarted","Data":"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"}
Oct 30 22:34:37.247756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:37.247720    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" event={"ID":"258c7c37-fefd-4260-b730-cf12816855e2","Type":"ContainerStarted","Data":"61efafb5db693b9a39b3b2a9216315e140d1e7a2d17179afd69b3eb5f8ac547c"}
Oct 30 22:34:37.363766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:37.363436    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" podStartSLOduration=1.363419864 podStartE2EDuration="1.363419864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:37.362729145 +0000 UTC m=+3130.451466961" watchObservedRunningTime="2026-10-30 22:34:37.363419864 +0000 UTC m=+3130.452157668"
Oct 30 22:34:37.573920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:37.573850    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:41.792615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:41.792576    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:42.097634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:42.097543    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:42.259035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:42.258998    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:42.414410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:42.414383    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.930026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:44.929989    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:46.046996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:46.046951    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"]
Oct 30 22:34:46.095354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:46.095321    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ef73e4-pqgs2"]
Oct 30 22:34:46.573980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:46.573949    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:46.586715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:46.586683    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:47.019050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:47.019016    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2493766f-43ed-4f88-a24c-2e083b33b45b" path="/var/lib/kubelet/pods/2493766f-43ed-4f88-a24c-2e083b33b45b/volumes"
Oct 30 22:34:47.121117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:47.121058    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:47.276361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:47.276284    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:47.321051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:47.321026    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:34:49.593113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:49.593077    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:49.786679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:49.786642    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:49.996132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:49.996095    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:34:50.001226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.001209    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.045842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.045818    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"]
Oct 30 22:34:50.069346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.069323    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66928bd-tm697"]
Oct 30 22:34:50.112377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.112339    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.112711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.112689    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9v4\" (UniqueName: \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.119798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.118599    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:34:50.119798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.119120    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9651e9-dockercfg-4p5bv\""
Oct 30 22:34:50.120076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.120060    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9651e9\""
Oct 30 22:34:50.214350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.214317    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9v4\" (UniqueName: \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.214538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.214494    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.215277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.215246    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.271599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.271536    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9v4\" (UniqueName: \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\") pod \"payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.314963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.314739    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:50.522324 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:34:50.522247    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:34:50.688704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:50.688675    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:34:50.689996 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:34:50.689969    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30886629_2c6b_4514_a9c3_e9b46d5dafc5.slice/crio-a535a8279799e67a0313dad0743cf148a873aad7991a093eb58af632ca558f25 WatchSource:0}: Error finding container a535a8279799e67a0313dad0743cf148a873aad7991a093eb58af632ca558f25: Status 404 returned error can't find the container with id a535a8279799e67a0313dad0743cf148a873aad7991a093eb58af632ca558f25
Oct 30 22:34:51.019789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:51.019742    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38641418-2105-4296-b549-05083e61662d" path="/var/lib/kubelet/pods/38641418-2105-4296-b549-05083e61662d/volumes"
Oct 30 22:34:51.334564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:51.334467    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" event={"ID":"30886629-2c6b-4514-a9c3-e9b46d5dafc5","Type":"ContainerStarted","Data":"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"}
Oct 30 22:34:51.334564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:51.334531    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" event={"ID":"30886629-2c6b-4514-a9c3-e9b46d5dafc5","Type":"ContainerStarted","Data":"a535a8279799e67a0313dad0743cf148a873aad7991a093eb58af632ca558f25"}
Oct 30 22:34:51.334866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:51.334672    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:34:51.447993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:51.447948    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" podStartSLOduration=2.447936133 podStartE2EDuration="2.447936133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:51.447461203 +0000 UTC m=+3144.536199007" watchObservedRunningTime="2026-10-30 22:34:51.447936133 +0000 UTC m=+3144.536673936"
Oct 30 22:34:52.324583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:52.324552    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:57.044607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:57.044574    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"]
Oct 30 22:34:57.093891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:57.093854    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69330c9c-vzq75"]
Oct 30 22:34:57.159667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:57.159632    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:59.022498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:34:59.022461    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593b2375-c50d-4de0-aca8-57a8df512166" path="/var/lib/kubelet/pods/593b2375-c50d-4de0-aca8-57a8df512166/volumes"
Oct 30 22:35:01.051338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:01.051299    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"]
Oct 30 22:35:01.100656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:01.100623    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"]
Oct 30 22:35:01.103048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:01.103030    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbff259f-lwgtr"]
Oct 30 22:35:01.105104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:01.105079    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc1c26f-lwhwf"]
Oct 30 22:35:01.998274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:01.998234    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:02.343399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:02.343319    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:35:03.020372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:03.020333    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cc7018-88b6-4d54-b8c8-6c4322adaa5b" path="/var/lib/kubelet/pods/10cc7018-88b6-4d54-b8c8-6c4322adaa5b/volumes"
Oct 30 22:35:03.020946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:03.020928    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c464fcba-86e8-4ea4-b18b-43796dea3f4b" path="/var/lib/kubelet/pods/c464fcba-86e8-4ea4-b18b-43796dea3f4b/volumes"
Oct 30 22:35:06.803766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:06.803732    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.648086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:11.648055    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:16.457517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:16.457469    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:16.699048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:16.699012    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:16.893355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:16.893320    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:20.523879 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:35:20.523830    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:35:21.500247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:21.500210    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:26.046293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:26.046261    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"]
Oct 30 22:35:26.095426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:26.095397    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf04f3f-d82lz"]
Oct 30 22:35:26.360310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:26.360234    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:27.020735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:27.020698    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c39f1e6-7eae-485d-9aa6-fe325afe7c2c" path="/var/lib/kubelet/pods/3c39f1e6-7eae-485d-9aa6-fe325afe7c2c/volumes"
Oct 30 22:35:29.151314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:29.151279    4410 scope.go:122] "RemoveContainer" containerID="05cf53b85f55ac38423335aee60ce9fee3dd7cc65a8ee7cda75586944dd67173"
Oct 30 22:35:29.160410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:29.160392    4410 scope.go:122] "RemoveContainer" containerID="5955cd17acfc2e9c3cf5916c7d4d598bb4f03c458d41912b38afe375d332b180"
Oct 30 22:35:29.168880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:29.168859    4410 scope.go:122] "RemoveContainer" containerID="352902ee61f1c34da1758277df347886d6e8a55a06778d972dc1a4ed3c3378e2"
Oct 30 22:35:29.176768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:29.176752    4410 scope.go:122] "RemoveContainer" containerID="1b10d30de0100a4e19a930ff77f672a212040427ab6bf87a7ef013404c7be92d"
Oct 30 22:35:31.250915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:31.250883    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:31.736962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:31.736900    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:35.319233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:35.319198    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:36.708455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:36.708422    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:41.432526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:41.432448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:41.789481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:41.789380    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.777180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:45.777149    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.177703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.177674    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.636822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.636719    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.689755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.689724    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:35:46.696585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.693547    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.762668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.762637    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0e450a-openshift-default-dockercfg-gsxpp\""
Oct 30 22:35:46.790172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.790143    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:35:46.819368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.819338    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:46.838588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838548    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.838823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838602    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.838823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838649    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.838823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838734    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.839031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838864    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.839031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838918    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbp5z\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.839031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.838974    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.839031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.839020    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.839260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.839078    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940517    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940568    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940610    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940685    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940785    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.940829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940819    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbp5z\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940878    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940923    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.940996    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.941172    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.941247    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.941285    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.941330    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.941746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.941721    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.943360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.943339    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.943691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.943671    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.997417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.997359    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:46.997417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:46.997385    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbp5z\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\") pod \"e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") " pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:47.008943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.008919    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:47.078230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.076688    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.459397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.459361    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:35:47.460075 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:35:47.460044    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4359549_ca53_4e0d_96f0_ad450a2d74c7.slice/crio-d2e35c8c82957b63169492f1a00d085f71eaf7901e28bbf786d6302dfdcfeeab WatchSource:0}: Error finding container d2e35c8c82957b63169492f1a00d085f71eaf7901e28bbf786d6302dfdcfeeab: Status 404 returned error can't find the container with id d2e35c8c82957b63169492f1a00d085f71eaf7901e28bbf786d6302dfdcfeeab
Oct 30 22:35:47.462917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.462879    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:35:47.463002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.462962    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:35:47.463046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.463003    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:35:47.676786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.676738    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerStarted","Data":"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"}
Oct 30 22:35:47.676958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.676821    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerStarted","Data":"d2e35c8c82957b63169492f1a00d085f71eaf7901e28bbf786d6302dfdcfeeab"}
Oct 30 22:35:47.795764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:47.795657    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podStartSLOduration=1.795639188 podStartE2EDuration="1.795639188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:47.792757402 +0000 UTC m=+3200.881495240" watchObservedRunningTime="2026-10-30 22:35:47.795639188 +0000 UTC m=+3200.884376993"
Oct 30 22:35:48.010053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:48.010023    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:48.015762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:48.015737    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:48.683245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:48.683198    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:48.684128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:48.684109    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:35:50.521667 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:35:50.521625    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:35:50.537516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:50.537490    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.757197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:50.757168    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:51.914539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:51.914499    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:55.234194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.234151    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:35:55.238078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.238057    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.332126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.332094    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:55.336331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336305    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.336441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336342    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.336441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336375    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.336544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336494    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpx5d\" (UniqueName: \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.336544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336515    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.336654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.336634    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.341416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.341398    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6c788d\"/\"kube-root-ca.crt\""
Oct 30 22:35:55.341656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.341640    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6c788d\"/\"openshift-service-ca.crt\""
Oct 30 22:35:55.341733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.341647    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6c788d\"/\"default-dockercfg-fwjn5\""
Oct 30 22:35:55.341733 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.341658    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6c788d\"/\"auth-test-model-6c788d-kserve-self-signed-certs\""
Oct 30 22:35:55.343900 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.343879    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:35:55.438097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438063    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpx5d\" (UniqueName: \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438105    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438174    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438257    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438280    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438339    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438641    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438655    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.438812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.438759    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.440542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.440519    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.440838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.440819    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.495222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.495155    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpx5d\" (UniqueName: \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\") pod \"auth-test-model-6c788d-kserve-f4456f6b5-g6jmf\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") " pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.550145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.550114    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:55.924917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:55.924891    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:35:55.926536 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:35:55.926507    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09324fa8_a2a1_4eb7_ab38_8df1ecae4467.slice/crio-b2f4bd508967ec8712a38ae9f386d9983fec5a4bc5bcc252612619a6d9558d96 WatchSource:0}: Error finding container b2f4bd508967ec8712a38ae9f386d9983fec5a4bc5bcc252612619a6d9558d96: Status 404 returned error can't find the container with id b2f4bd508967ec8712a38ae9f386d9983fec5a4bc5bcc252612619a6d9558d96
Oct 30 22:35:56.737694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:56.737659    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" event={"ID":"09324fa8-a2a1-4eb7-ab38-8df1ecae4467","Type":"ContainerStarted","Data":"478d30cf500027a02054b305b809af3da16b4f28885d031c712aa0cc983d11fd"}
Oct 30 22:35:56.737694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:56.737705    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" event={"ID":"09324fa8-a2a1-4eb7-ab38-8df1ecae4467","Type":"ContainerStarted","Data":"b2f4bd508967ec8712a38ae9f386d9983fec5a4bc5bcc252612619a6d9558d96"}
Oct 30 22:35:56.859232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:56.859177    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" podStartSLOduration=1.859164137 podStartE2EDuration="1.859164137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:56.856197581 +0000 UTC m=+3209.944935384" watchObservedRunningTime="2026-10-30 22:35:56.859164137 +0000 UTC m=+3209.947901940"
Oct 30 22:35:56.929448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:56.929406    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:57.551169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:57.551142    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:35:59.599315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:59.599280    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:59.789227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:35:59.789194    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.710280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:01.710217    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:05.550985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:05.550946    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:36:05.563297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:05.563268    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:36:05.818540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:05.818466    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:36:06.553567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:06.553534    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:07.609300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:07.609260    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:07.771125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:07.771091    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.379167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:11.379132    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:17.329249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:17.329199    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:20.522632 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:36:20.522598    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:36:20.984850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:20.984811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:25.758504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:25.758464    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:29.214119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.214081    4410 scope.go:122] "RemoveContainer" containerID="9487450bd534667a3ef8de530e170eebf0c56f1ced14c49d879713b37dd143b5"
Oct 30 22:36:29.223647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.223619    4410 scope.go:122] "RemoveContainer" containerID="683df11f57b8dfac7cc8cdedab4d742c72f3d92f8d74bc23c0d0b51046881b62"
Oct 30 22:36:29.232158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.231869    4410 scope.go:122] "RemoveContainer" containerID="3d4e9d00e2d6c2185156a1b709c1894cef1fde856f5da36a9af7caa9e9af83fb"
Oct 30 22:36:29.240409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.240390    4410 scope.go:122] "RemoveContainer" containerID="1e6e6e722671b5c5653ac22cdde28884842c947a86bab13add086ac3d0bb77b6"
Oct 30 22:36:29.248271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.248253    4410 scope.go:122] "RemoveContainer" containerID="efeb624bdea78ed65fcea12c33fd638d7d077b9b646482b8476bcec3b69592df"
Oct 30 22:36:29.255836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.255819    4410 scope.go:122] "RemoveContainer" containerID="03b7c0dff631b9c682b090173b9eab35d2f0bbdcccfd45cb8647f6108817a66c"
Oct 30 22:36:29.263320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.263302    4410 scope.go:122] "RemoveContainer" containerID="1992052197e1c6bd0f9723228f50b1eaa1de5e0f27105a0b850da6d0a159faef"
Oct 30 22:36:29.271212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:29.271136    4410 scope.go:122] "RemoveContainer" containerID="c4c01c736cb129d623d3579d0bed2b53a166223b0d6ba2247e8cb2ec85ba0e76"
Oct 30 22:36:30.648979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:30.648939    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:31.434030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:31.433993    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:32.085486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:32.085451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:35.941690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:35.941652    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:36.616929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:36.616890    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:40.949755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:40.949710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:41.046048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:41.046011    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"]
Oct 30 22:36:41.094913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:41.094886    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b912aa-xxs9f"]
Oct 30 22:36:43.020027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:43.019995    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cd5595-07e4-4e4f-b9c0-3510c02f4183" path="/var/lib/kubelet/pods/a5cd5595-07e4-4e4f-b9c0-3510c02f4183/volumes"
Oct 30 22:36:45.733862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:45.733825    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:47.284860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:47.284822    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:50.498838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:50.498804    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:50.522925 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:36:50.522886    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:36:55.307546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:55.307512    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:58.110626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:36:58.110566    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.147520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:00.147487    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.339501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:00.339462    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.508523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:00.508442    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:05.577701 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:05.577664    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:10.463813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:10.463695    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:15.086613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:15.086561    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:15.559471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:15.559434    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:18.803175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:18.803121    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:19.027170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:19.027129    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:20.382404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:20.382346    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:20.521631 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:37:20.521597    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:37:25.254210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.254176    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:25.320859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.320823    4410 generic.go:356] "Generic (PLEG): container finished" podID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerID="c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd" exitCode=0
Oct 30 22:37:25.321077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.320897    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerDied","Data":"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"}
Oct 30 22:37:25.321370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.321351    4410 scope.go:122] "RemoveContainer" containerID="c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"
Oct 30 22:37:25.322945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.322916    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:37:25.323040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.322987    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:37:25.323040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:25.323018    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:37:26.329111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:26.329077    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerStarted","Data":"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"}
Oct 30 22:37:27.009119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:27.009085    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:37:27.009300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:27.009230    4410 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:37:27.009300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:27.009272    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:37:28.009984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:28.009948    4410 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:37:28.010430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:28.010038    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:37:29.009654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:29.009618    4410 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:37:29.009871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:29.009678    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:37:29.322238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:29.322158    4410 scope.go:122] "RemoveContainer" containerID="829e2663facce2630e23b36ed6589f8326ea076226a6f6410c95d34dc7262582"
Oct 30 22:37:30.010284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:30.010250    4410 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:37:30.010463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:30.010313    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:37:30.142611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:30.142578    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:31.009423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:31.009388    4410 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:37:31.009889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:31.009443    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:37:32.014425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:32.014395    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:37:32.014926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:32.014862    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:37:32.015573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:32.015553    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:37:32.422183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:32.422150    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:34.932175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:34.932132    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:39.801730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:39.801692    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:44.645702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:44.645665    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.436466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:49.436428    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.522084 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:37:50.522044    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:37:52.463322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:52.463283    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:54.255335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:54.255298    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:59.039619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:37:59.039584    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:00.048440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:00.048397    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"]
Oct 30 22:38:00.097080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:00.097041    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce0314d-52tmh"]
Oct 30 22:38:01.022027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:01.021986    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cd8993-d4f9-4f51-80c6-f3ceedd77be7" path="/var/lib/kubelet/pods/95cd8993-d4f9-4f51-80c6-f3ceedd77be7/volumes"
Oct 30 22:38:03.822841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:03.822795    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:08.643922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:08.643881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:11.512501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:11.512451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:13.551450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:13.551407    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:14.134392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:14.134361    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:38:14.134625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:14.134592    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" containerID="cri-o://7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c" gracePeriod=30
Oct 30 22:38:18.360756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:18.360712    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:18.926164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:18.926124    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:38:18.926346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:18.926197    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:38:19.404490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.404463    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:38:19.490665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490630    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.490868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490673    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.490868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490704    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.490868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490763    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.490868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490821    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490859    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490955    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.490990    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491029    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491056    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:19.491161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491074    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\") pod \"2b3df4a2-a503-4563-a096-f81435961847\" (UID: \"2b3df4a2-a503-4563-a096-f81435961847\") "
Oct 30 22:38:19.491487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491268    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:19.491487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491282    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:19.491592 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491535    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:19.491674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491653    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.491732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491680    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.491732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491695    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.491732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491710    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3df4a2-a503-4563-a096-f81435961847-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.491732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.491725    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.493242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.493202    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:19.493393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.493375    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:19.493514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.493500    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:19.493575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.493543    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx" pod "2b3df4a2-a503-4563-a096-f81435961847" (UID: "2b3df4a2-a503-4563-a096-f81435961847"). InnerVolumeSpecName "kube-api-access-924xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:19.592647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.592576    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.592647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.592607    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3df4a2-a503-4563-a096-f81435961847-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.592647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.592621    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924xx\" (UniqueName: \"kubernetes.io/projected/2b3df4a2-a503-4563-a096-f81435961847-kube-api-access-924xx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.592647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.592632    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3df4a2-a503-4563-a096-f81435961847-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.685281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.685246    4410 generic.go:356] "Generic (PLEG): container finished" podID="2b3df4a2-a503-4563-a096-f81435961847" containerID="7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c" exitCode=0
Oct 30 22:38:19.685438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.685300    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"
Oct 30 22:38:19.685438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.685327    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" event={"ID":"2b3df4a2-a503-4563-a096-f81435961847","Type":"ContainerDied","Data":"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"}
Oct 30 22:38:19.685438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.685404    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7" event={"ID":"2b3df4a2-a503-4563-a096-f81435961847","Type":"ContainerDied","Data":"9e4d9d9426e497098585b513c7d660b526935a386cb6b1a1759b3a3d8a22dd19"}
Oct 30 22:38:19.685438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.685426    4410 scope.go:122] "RemoveContainer" containerID="7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"
Oct 30 22:38:19.695769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.695749    4410 scope.go:122] "RemoveContainer" containerID="7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"
Oct 30 22:38:19.696047 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:38:19.696027    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c\": container with ID starting with 7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c not found: ID does not exist" containerID="7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"
Oct 30 22:38:19.696108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.696055    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c"} err="failed to get container status \"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c\": rpc error: code = NotFound desc = could not find container \"7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c\": container with ID starting with 7ccaab0321a3123a34b847aadf68300efb9e9fddeb1bdcbbd6e0ed7f7c4f0a4c not found: ID does not exist"
Oct 30 22:38:19.858026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.857950    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:38:19.863381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:19.863356    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-gx4g7"]
Oct 30 22:38:20.523285 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:38:20.523248    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:38:21.019563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:21.019532    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3df4a2-a503-4563-a096-f81435961847" path="/var/lib/kubelet/pods/2b3df4a2-a503-4563-a096-f81435961847/volumes"
Oct 30 22:38:23.622386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:23.622346    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.786135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:23.786097    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:28.449878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:28.449846    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:29.344653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:29.344619    4410 scope.go:122] "RemoveContainer" containerID="0502b216b2d4803cc04fb11cf3c557a76a850cf02508389ba3631369bebbdd60"
Oct 30 22:38:29.353637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:29.353611    4410 scope.go:122] "RemoveContainer" containerID="e9ce67e74a0dacc38b3b08512b5a5cbf9616d6cffd07483ec3d33141af1221df"
Oct 30 22:38:29.363131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:29.363111    4410 scope.go:122] "RemoveContainer" containerID="1124127ca4fd50d9d2c92feb3c15ac4b66a5f7cb4e95a431dfcba41c5453c464"
Oct 30 22:38:29.371987 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:29.371970    4410 scope.go:122] "RemoveContainer" containerID="10d6f8d70eeb8e5af5023da1b18478c456403ed024e2eb1b9a730f5bbe64a9ba"
Oct 30 22:38:32.763292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:32.763257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:32.962377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:32.962340    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:33.235564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:33.235520    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:38.059156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:38.059120    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:42.853504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:42.853409    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.905852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:43.905812    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:47.763542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:47.763510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:49.513917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.513880    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"]
Oct 30 22:38:49.514387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.514368    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy"
Oct 30 22:38:49.514387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.514380    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy"
Oct 30 22:38:49.514489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.514477    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3df4a2-a503-4563-a096-f81435961847" containerName="istio-proxy"
Oct 30 22:38:49.517463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.517446    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.619802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.619742    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:49.619802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.619753    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:49.620041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.619767    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:38:49.620041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.619851    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:49.620258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.620238    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.620352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.620279    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.620411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.620368    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rl8\" (UniqueName: \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.622588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.622563    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"]
Oct 30 22:38:49.721637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.721606    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.721829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.721656    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.721829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.721693    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rl8\" (UniqueName: \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.722406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.722386    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.724132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.724105    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.779541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.779470    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rl8\" (UniqueName: \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\") pod \"maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:49.830155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:49.830124    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:50.210225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:50.210200    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"]
Oct 30 22:38:50.212496 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:38:50.212470    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442bc603_83a1_4d8e_8c09_de57462347cc.slice/crio-3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654 WatchSource:0}: Error finding container 3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654: Status 404 returned error can't find the container with id 3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654
Oct 30 22:38:50.521375 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:38:50.521300    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:38:50.873035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:50.872955    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8" event={"ID":"442bc603-83a1-4d8e-8c09-de57462347cc","Type":"ContainerStarted","Data":"490581ae856e279bff0fcedfc9d3d46f5d25c16fe4ce9e5709f0ae2fddb221e2"}
Oct 30 22:38:50.873035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:50.873033    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8" event={"ID":"442bc603-83a1-4d8e-8c09-de57462347cc","Type":"ContainerStarted","Data":"3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654"}
Oct 30 22:38:51.009646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:51.009595    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8" podStartSLOduration=2.009578906 podStartE2EDuration="2.009578906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:51.00830104 +0000 UTC m=+3384.097038907" watchObservedRunningTime="2026-10-30 22:38:51.009578906 +0000 UTC m=+3384.098316713"
Oct 30 22:38:51.060816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:51.060750    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"]
Oct 30 22:38:51.063022 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:51.063000    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d59290-56sd8"]
Oct 30 22:38:51.881526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:51.881489    4410 generic.go:356] "Generic (PLEG): container finished" podID="442bc603-83a1-4d8e-8c09-de57462347cc" containerID="490581ae856e279bff0fcedfc9d3d46f5d25c16fe4ce9e5709f0ae2fddb221e2" exitCode=0
Oct 30 22:38:51.882078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:51.881533    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8" event={"ID":"442bc603-83a1-4d8e-8c09-de57462347cc","Type":"ContainerDied","Data":"490581ae856e279bff0fcedfc9d3d46f5d25c16fe4ce9e5709f0ae2fddb221e2"}
Oct 30 22:38:53.021204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.021172    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731bcc6a-7232-417f-ac57-5c3df3f41986" path="/var/lib/kubelet/pods/731bcc6a-7232-417f-ac57-5c3df3f41986/volumes"
Oct 30 22:38:53.044618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.044596    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:53.045171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.045146    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:53.163302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.163273    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\") pod \"442bc603-83a1-4d8e-8c09-de57462347cc\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") "
Oct 30 22:38:53.163479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.163386    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\" (UniqueName: \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\") pod \"442bc603-83a1-4d8e-8c09-de57462347cc\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") "
Oct 30 22:38:53.163479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.163423    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\") pod \"442bc603-83a1-4d8e-8c09-de57462347cc\" (UID: \"442bc603-83a1-4d8e-8c09-de57462347cc\") "
Oct 30 22:38:53.163622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.163600    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca" pod "442bc603-83a1-4d8e-8c09-de57462347cc" (UID: "442bc603-83a1-4d8e-8c09-de57462347cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:53.163957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.163933    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:53.165589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.165560    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8" pod "442bc603-83a1-4d8e-8c09-de57462347cc" (UID: "442bc603-83a1-4d8e-8c09-de57462347cc"). InnerVolumeSpecName "kube-api-access-62rl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:53.165880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.165861    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca" pod "442bc603-83a1-4d8e-8c09-de57462347cc" (UID: "442bc603-83a1-4d8e-8c09-de57462347cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:53.265525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.265495    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62rl8\" (UniqueName: \"kubernetes.io/projected/442bc603-83a1-4d8e-8c09-de57462347cc-kube-api-access-62rl8\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:53.265525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.265524    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442bc603-83a1-4d8e-8c09-de57462347cc-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:53.893939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.893914    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"
Oct 30 22:38:53.894122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.893910    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8" event={"ID":"442bc603-83a1-4d8e-8c09-de57462347cc","Type":"ContainerDied","Data":"3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654"}
Oct 30 22:38:53.894122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:53.894044    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e953d9b0e62aff3e150f7c47b559f99705dcb635b3fe51da755789d54da5654"
Oct 30 22:38:57.849719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:38:57.849681    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:01.148166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:01.148128    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:39:01.440370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:01.440288    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:01.625208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:01.625171    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:02.674562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:02.674526    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:06.149295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.149253    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" podUID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" containerName="payload-pre-processing" containerID="cri-o://cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594" gracePeriod=25
Oct 30 22:39:06.400076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.400016    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:39:06.506025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.505993    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\" (UniqueName: \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\") pod \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") "
Oct 30 22:39:06.506180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.506052    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\") pod \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\" (UID: \"30886629-2c6b-4514-a9c3-e9b46d5dafc5\") "
Oct 30 22:39:06.506423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.506392    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume" pod "30886629-2c6b-4514-a9c3-e9b46d5dafc5" (UID: "30886629-2c6b-4514-a9c3-e9b46d5dafc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:06.508079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.508060    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4" pod "30886629-2c6b-4514-a9c3-e9b46d5dafc5" (UID: "30886629-2c6b-4514-a9c3-e9b46d5dafc5"). InnerVolumeSpecName "kube-api-access-4t9v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:06.607816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.607767    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9v4\" (UniqueName: \"kubernetes.io/projected/30886629-2c6b-4514-a9c3-e9b46d5dafc5-kube-api-access-4t9v4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:06.607816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.607815    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30886629-2c6b-4514-a9c3-e9b46d5dafc5-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:06.972527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.972498    4410 generic.go:356] "Generic (PLEG): container finished" podID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" containerID="cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594" exitCode=0
Oct 30 22:39:06.972703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.972541    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"
Oct 30 22:39:06.972703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.972577    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" event={"ID":"30886629-2c6b-4514-a9c3-e9b46d5dafc5","Type":"ContainerDied","Data":"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"}
Oct 30 22:39:06.972703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.972628    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf" event={"ID":"30886629-2c6b-4514-a9c3-e9b46d5dafc5","Type":"ContainerDied","Data":"a535a8279799e67a0313dad0743cf148a873aad7991a093eb58af632ca558f25"}
Oct 30 22:39:06.972703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.972655    4410 scope.go:122] "RemoveContainer" containerID="cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"
Oct 30 22:39:06.982893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.982872    4410 scope.go:122] "RemoveContainer" containerID="cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"
Oct 30 22:39:06.983140 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:39:06.983117    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594\": container with ID starting with cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594 not found: ID does not exist" containerID="cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"
Oct 30 22:39:06.983213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:06.983148    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594"} err="failed to get container status \"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594\": rpc error: code = NotFound desc = could not find container \"cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594\": container with ID starting with cdd695b8c9d6eccb41a408ea4563c0c7055b207989108c5f79d0da01aa751594 not found: ID does not exist"
Oct 30 22:39:07.140959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:07.140927    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:39:07.190265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:07.190237    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9651e9-7974fc89c9-pt9xf"]
Oct 30 22:39:07.438525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:07.438491    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:09.019425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:09.019396    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" path="/var/lib/kubelet/pods/30886629-2c6b-4514-a9c3-e9b46d5dafc5/volumes"
Oct 30 22:39:12.329608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:12.329567    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:16.312302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:16.312247    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.146736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:17.146695    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:18.582872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:18.582827    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:20.522297 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:39:20.522258    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:39:21.440891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:21.440853    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:22.981295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:22.981254    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:23.198379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:23.198341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:23.362088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:23.361995    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:23.585065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:23.585032    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"]
Oct 30 22:39:23.633699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:23.633635    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9651e9-sk8m8"]
Oct 30 22:39:24.016794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:24.016741    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.019581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:25.019547    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442bc603-83a1-4d8e-8c09-de57462347cc" path="/var/lib/kubelet/pods/442bc603-83a1-4d8e-8c09-de57462347cc/volumes"
Oct 30 22:39:25.729398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:25.729359    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:39:25.729583 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:25.729545    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy" containerID="cri-o://9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f" gracePeriod=30
Oct 30 22:39:28.374663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:28.374618    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:29.053183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:29.053148    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:29.414688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:29.414659    4410 scope.go:122] "RemoveContainer" containerID="49c1a4620f3b634092cf841a0503b3a3e6c2c54049c857f01646b8948e74faef"
Oct 30 22:39:30.069162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:30.069130    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:30.993272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:30.993212    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:39:31.079123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079092    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079131    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079219    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079252    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079284    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079335    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079390    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.079509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079463    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079484    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.079708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079510    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079575    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\") pod \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\" (UID: \"fd8cd2ed-2438-4b43-a61d-d0cc5531c754\") "
Oct 30 22:39:31.079708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079625    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.079930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079805    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:31.079930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.079823    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.081241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.081217    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.081420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.081404    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.081885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.081856    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.082027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.082012    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.082147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.082122    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.082265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.082248    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:31.084922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.084897    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.092436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.092395    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:31.092880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.092857    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk" pod "fd8cd2ed-2438-4b43-a61d-d0cc5531c754" (UID: "fd8cd2ed-2438-4b43-a61d-d0cc5531c754"). InnerVolumeSpecName "kube-api-access-sj9pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:31.109366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.109343    4410 generic.go:356] "Generic (PLEG): container finished" podID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerID="9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f" exitCode=0
Oct 30 22:39:31.109473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.109384    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" event={"ID":"fd8cd2ed-2438-4b43-a61d-d0cc5531c754","Type":"ContainerDied","Data":"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"}
Oct 30 22:39:31.109473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.109398    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"
Oct 30 22:39:31.109473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.109462    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx" event={"ID":"fd8cd2ed-2438-4b43-a61d-d0cc5531c754","Type":"ContainerDied","Data":"7c19ddfd37a0f7dd400ecd0027753dacf25ea98751d50ab12243c41b420081a2"}
Oct 30 22:39:31.109645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.109493    4410 scope.go:122] "RemoveContainer" containerID="9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"
Oct 30 22:39:31.119149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.119131    4410 scope.go:122] "RemoveContainer" containerID="9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"
Oct 30 22:39:31.119421 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:39:31.119402    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f\": container with ID starting with 9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f not found: ID does not exist" containerID="9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"
Oct 30 22:39:31.119491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.119432    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f"} err="failed to get container status \"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f\": rpc error: code = NotFound desc = could not find container \"9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f\": container with ID starting with 9b8e393607106cd040ab737c65ed57949e451653ed4998d308721cea7b15178f not found: ID does not exist"
Oct 30 22:39:31.183744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.183712    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj9pk\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-kube-api-access-sj9pk\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.183744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.183739    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.183744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.183749    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.184004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.183758    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8cd2ed-2438-4b43-a61d-d0cc5531c754-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.270005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.269934    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:39:31.275753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:31.275728    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-c4knx"]
Oct 30 22:39:33.019238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:33.019197    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" path="/var/lib/kubelet/pods/fd8cd2ed-2438-4b43-a61d-d0cc5531c754/volumes"
Oct 30 22:39:33.074239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:33.074200    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:33.381577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:33.381499    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:35.321850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:35.321816    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:35.660168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:35.660128    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:36.313354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:36.313313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:36.941448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:36.941414    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:37.229819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:37.229710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:38.068540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:38.068502    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:42.885983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:42.885940    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.034435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:47.034403    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.268223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:47.268188    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.513527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:47.513466    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.813851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:47.813729    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:50.521933 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:39:50.521896    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:39:52.754649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:52.754617    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:57.590714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:39:57.590680    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:02.378114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:02.378062    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.151709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:07.151666    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:08.078721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.078692    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"]
Oct 30 22:40:08.079188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079174    4410 cpu_manager.go:414] "containerMap: removing container" podUID="442bc603-83a1-4d8e-8c09-de57462347cc" containerName="revoke-keys"
Oct 30 22:40:08.079243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079190    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442bc603-83a1-4d8e-8c09-de57462347cc" containerName="revoke-keys"
Oct 30 22:40:08.079243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079200    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy"
Oct 30 22:40:08.079243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079206    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy"
Oct 30 22:40:08.079243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079217    4410 cpu_manager.go:414] "containerMap: removing container" podUID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" containerName="payload-pre-processing"
Oct 30 22:40:08.079243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079222    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" containerName="payload-pre-processing"
Oct 30 22:40:08.079400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079295    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="442bc603-83a1-4d8e-8c09-de57462347cc" containerName="revoke-keys"
Oct 30 22:40:08.079400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079305    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8cd2ed-2438-4b43-a61d-d0cc5531c754" containerName="istio-proxy"
Oct 30 22:40:08.079400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.079311    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="30886629-2c6b-4514-a9c3-e9b46d5dafc5" containerName="payload-pre-processing"
Oct 30 22:40:08.082457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.082440    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.177455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.177426    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxxg\" (UniqueName: \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.177884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.177603    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.177884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.177637    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.184493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.184473    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:08.184493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.184473    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:08.184655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.184473    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:08.184655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.184506    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:40:08.187037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.187020    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"]
Oct 30 22:40:08.279105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.279075    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.279264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.279113    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.279264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.279224    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxxg\" (UniqueName: \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.279884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.279859    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.281438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.281416    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.336752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.336690    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxxg\" (UniqueName: \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\") pod \"maas-api-revoke-sub-65ffec2f-4964k\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.395133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.395104    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:08.776356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.776328    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"]
Oct 30 22:40:08.779110 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:40:08.779072    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b2b61f_c7cf_4f8d_8269_5e3c52ddce84.slice/crio-6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8 WatchSource:0}: Error finding container 6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8: Status 404 returned error can't find the container with id 6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8
Oct 30 22:40:08.813247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.813215    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:08.969657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:08.969624    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:09.314892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:09.314805    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k" event={"ID":"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84","Type":"ContainerStarted","Data":"5dcd26efbcfb1bfb6281df8f04adb7259151c82ce6d2cc5245cacc2f5269e3aa"}
Oct 30 22:40:09.314892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:09.314871    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k" event={"ID":"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84","Type":"ContainerStarted","Data":"6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8"}
Oct 30 22:40:09.430642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:09.430585    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k" podStartSLOduration=1.430572213 podStartE2EDuration="1.430572213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:09.42873903 +0000 UTC m=+3462.517476836" watchObservedRunningTime="2026-10-30 22:40:09.430572213 +0000 UTC m=+3462.519310015"
Oct 30 22:40:10.322153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:10.322112    4410 generic.go:356] "Generic (PLEG): container finished" podID="21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" containerID="5dcd26efbcfb1bfb6281df8f04adb7259151c82ce6d2cc5245cacc2f5269e3aa" exitCode=0
Oct 30 22:40:10.322562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:10.322189    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k" event={"ID":"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84","Type":"ContainerDied","Data":"5dcd26efbcfb1bfb6281df8f04adb7259151c82ce6d2cc5245cacc2f5269e3aa"}
Oct 30 22:40:11.467336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.467311    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:11.515082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.515041    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\") pod \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") "
Oct 30 22:40:11.515222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.515114    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\" (UniqueName: \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\") pod \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") "
Oct 30 22:40:11.515293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.515281    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\") pod \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\" (UID: \"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84\") "
Oct 30 22:40:11.515668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.515637    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca" pod "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" (UID: "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:11.517200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.517177    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg" pod "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" (UID: "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84"). InnerVolumeSpecName "kube-api-access-crxxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:11.517305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.517198    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca" pod "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" (UID: "21b2b61f-c7cf-4f8d-8269-5e3c52ddce84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:11.617231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.617156    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.617231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.617193    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxxg\" (UniqueName: \"kubernetes.io/projected/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-kube-api-access-crxxg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.617231 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:11.617212    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:12.153992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:12.153960    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:12.334343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:12.334316    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"
Oct 30 22:40:12.334522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:12.334352    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k" event={"ID":"21b2b61f-c7cf-4f8d-8269-5e3c52ddce84","Type":"ContainerDied","Data":"6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8"}
Oct 30 22:40:12.334522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:12.334405    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6593b8a74bde1db71928cd4859e1fb6c9358667d3c72bfc699fab03ab7753cf8"
Oct 30 22:40:17.038530 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:17.038499    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:18.818184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.814599    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"]
Oct 30 22:40:18.818184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.815850    4410 cpu_manager.go:414] "containerMap: removing container" podUID="21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" containerName="revoke-keys"
Oct 30 22:40:18.818184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.815873    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" containerName="revoke-keys"
Oct 30 22:40:18.818184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.816123    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" containerName="revoke-keys"
Oct 30 22:40:18.820570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.820530    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.875556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.875528    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"]
Oct 30 22:40:18.897901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.897874    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.898057 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.898037    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcxz\" (UniqueName: \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.898175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.898154    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.921417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.921389    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:18.921535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.921444    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:18.921535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.921446    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:18.922520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.922504    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:40:18.999635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.999596    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.999829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.999697    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcxz\" (UniqueName: \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:18.999829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:18.999749    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:19.000251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:19.000227    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:19.002089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:19.002070    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:19.057746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:19.057716    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcxz\" (UniqueName: \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\") pod \"maas-api-revoke-sub-3c8175a5-8f2b5\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:19.134895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:19.134836    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:19.512461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:19.512435    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"]
Oct 30 22:40:19.514553 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:40:19.514520    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a8d7df_1138_41cc_a9c5_59182d6aac93.slice/crio-47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc WatchSource:0}: Error finding container 47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc: Status 404 returned error can't find the container with id 47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc
Oct 30 22:40:20.381681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:20.381639    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5" event={"ID":"c3a8d7df-1138-41cc-a9c5-59182d6aac93","Type":"ContainerStarted","Data":"7863e59587a75570a7d4f4d90533153a472cefe27a746a46c776e798bdb03663"}
Oct 30 22:40:20.382120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:20.381702    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5" event={"ID":"c3a8d7df-1138-41cc-a9c5-59182d6aac93","Type":"ContainerStarted","Data":"47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc"}
Oct 30 22:40:20.502403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:20.502355    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5" podStartSLOduration=2.502341195 podStartE2EDuration="2.502341195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:20.501126881 +0000 UTC m=+3473.589864687" watchObservedRunningTime="2026-10-30 22:40:20.502341195 +0000 UTC m=+3473.591079000"
Oct 30 22:40:20.523842 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:40:20.523810    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:40:21.389460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:21.389421    4410 generic.go:356] "Generic (PLEG): container finished" podID="c3a8d7df-1138-41cc-a9c5-59182d6aac93" containerID="7863e59587a75570a7d4f4d90533153a472cefe27a746a46c776e798bdb03663" exitCode=0
Oct 30 22:40:21.389893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:21.389501    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5" event={"ID":"c3a8d7df-1138-41cc-a9c5-59182d6aac93","Type":"ContainerDied","Data":"7863e59587a75570a7d4f4d90533153a472cefe27a746a46c776e798bdb03663"}
Oct 30 22:40:21.882811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:21.882717    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:22.527599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.527574    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:22.644832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.644769    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\") pod \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") "
Oct 30 22:40:22.645179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.644963    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\" (UniqueName: \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\") pod \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") "
Oct 30 22:40:22.645179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.645000    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\") pod \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\" (UID: \"c3a8d7df-1138-41cc-a9c5-59182d6aac93\") "
Oct 30 22:40:22.645329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.645266    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca" pod "c3a8d7df-1138-41cc-a9c5-59182d6aac93" (UID: "c3a8d7df-1138-41cc-a9c5-59182d6aac93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:22.645681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.645618    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:22.647731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.647704    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca" pod "c3a8d7df-1138-41cc-a9c5-59182d6aac93" (UID: "c3a8d7df-1138-41cc-a9c5-59182d6aac93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:22.647941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.647920    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz" pod "c3a8d7df-1138-41cc-a9c5-59182d6aac93" (UID: "c3a8d7df-1138-41cc-a9c5-59182d6aac93"). InnerVolumeSpecName "kube-api-access-mjcxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:22.746706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.746679    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcxz\" (UniqueName: \"kubernetes.io/projected/c3a8d7df-1138-41cc-a9c5-59182d6aac93-kube-api-access-mjcxz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:22.746706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:22.746704    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a8d7df-1138-41cc-a9c5-59182d6aac93-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:23.403214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:23.403182    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"
Oct 30 22:40:23.403393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:23.403205    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5" event={"ID":"c3a8d7df-1138-41cc-a9c5-59182d6aac93","Type":"ContainerDied","Data":"47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc"}
Oct 30 22:40:23.403393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:23.403261    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e7d8a8d5cb4f36aed34169e6058d5fcb1e4fe7bfe67c14d6a6a4791a774ebc"
Oct 30 22:40:26.751668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:26.751611    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:29.439784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:29.439741    4410 scope.go:122] "RemoveContainer" containerID="dfcd71ae2a1523f7650c26de0861222d76ccb4d9589ab59398553ca7a22bea09"
Oct 30 22:40:31.599829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:31.599749    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:33.415569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:33.415529    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:34.841286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:34.841223    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.441413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:36.441377    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:41.243116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:41.243077    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:46.167736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:46.167700    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:50.522361 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:40:50.522327    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:40:50.983689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:50.983658    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:55.907333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:40:55.907294    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:00.811861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:00.811819    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.598494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:05.598450    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.752560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:05.752521    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:10.376004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:10.375969    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:11.847628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:11.847589    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:41:11.874516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:11.874485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:15.240319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:15.240277    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:15.921667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:15.921630    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:16.254932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:16.254852    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:16.415541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:16.415511    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:17.540249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:17.540209    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:20.149687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:20.149654    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:20.521182 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:41:20.521103    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:41:21.427471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:21.427440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:24.949362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:24.949328    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:26.867884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:26.867847    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" podUID="258c7c37-fefd-4260-b730-cf12816855e2" containerName="main" containerID="cri-o://90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7" gracePeriod=45
Oct 30 22:41:27.139071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.139048    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:41:27.244482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244453    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244517    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244571    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244602    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244638    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\" (UniqueName: \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244723    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:27.244880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244752    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\") pod \"258c7c37-fefd-4260-b730-cf12816855e2\" (UID: \"258c7c37-fefd-4260-b730-cf12816855e2\") "
Oct 30 22:41:27.244880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244811    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:27.245100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.244964    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:27.245183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.245167    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.245247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.245184    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.245247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.245193    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.246829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.246806    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:27.246939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.246879    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "kube-api-access-rvzzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:27.246939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.246899    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm" pod "258c7c37-fefd-4260-b730-cf12816855e2" (UID: "258c7c37-fefd-4260-b730-cf12816855e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:27.346297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.346273    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258c7c37-fefd-4260-b730-cf12816855e2-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.346297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.346296    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258c7c37-fefd-4260-b730-cf12816855e2-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.346472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.346307    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzzl\" (UniqueName: \"kubernetes.io/projected/258c7c37-fefd-4260-b730-cf12816855e2-kube-api-access-rvzzl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.754702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.754667    4410 generic.go:356] "Generic (PLEG): container finished" podID="258c7c37-fefd-4260-b730-cf12816855e2" containerID="90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7" exitCode=0
Oct 30 22:41:27.754881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.754708    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"
Oct 30 22:41:27.754881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.754742    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" event={"ID":"258c7c37-fefd-4260-b730-cf12816855e2","Type":"ContainerDied","Data":"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"}
Oct 30 22:41:27.754881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.754799    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7" event={"ID":"258c7c37-fefd-4260-b730-cf12816855e2","Type":"ContainerDied","Data":"61efafb5db693b9a39b3b2a9216315e140d1e7a2d17179afd69b3eb5f8ac547c"}
Oct 30 22:41:27.754881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.754818    4410 scope.go:122] "RemoveContainer" containerID="90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"
Oct 30 22:41:27.764406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.764387    4410 scope.go:122] "RemoveContainer" containerID="90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"
Oct 30 22:41:27.764684 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:41:27.764663    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7\": container with ID starting with 90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7 not found: ID does not exist" containerID="90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"
Oct 30 22:41:27.764760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.764691    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7"} err="failed to get container status \"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7\": rpc error: code = NotFound desc = could not find container \"90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7\": container with ID starting with 90d6ca2fd200ee266b292604961e01bd9f1fa46c59389565de0f6798958821d7 not found: ID does not exist"
Oct 30 22:41:27.918289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.918263    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:41:27.922090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:27.922067    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a0116e/auth-test-model-a0116e-kserve-5d76d4cbfc-tplm7"]
Oct 30 22:41:29.020677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:29.020641    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258c7c37-fefd-4260-b730-cf12816855e2" path="/var/lib/kubelet/pods/258c7c37-fefd-4260-b730-cf12816855e2/volumes"
Oct 30 22:41:29.739304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:29.739226    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:30.180256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:30.180222    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:30.648503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:30.648470    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:41:30.648697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:30.648671    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" containerID="cri-o://0ac224a10cff11f7cbcc99c8bb499a964325f271ec1f07265869826c46df1810" gracePeriod=30
Oct 30 22:41:33.757272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:33.757237    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:34.655694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:34.655653    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:35.128929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.128854    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:41:35.128929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.128924    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:41:35.800510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.800473    4410 generic.go:356] "Generic (PLEG): container finished" podID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerID="0ac224a10cff11f7cbcc99c8bb499a964325f271ec1f07265869826c46df1810" exitCode=0
Oct 30 22:41:35.800698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.800542    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" event={"ID":"960c1ab3-72c2-45ed-9a7e-edc36297c59a","Type":"ContainerDied","Data":"0ac224a10cff11f7cbcc99c8bb499a964325f271ec1f07265869826c46df1810"}
Oct 30 22:41:35.904334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.904309    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:41:35.938923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.938858    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.938923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.938896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.938928    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939052    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939094    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939182    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939134    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.939283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939272    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.939498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939282    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939331    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.939498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939378    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\") pod \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\" (UID: \"960c1ab3-72c2-45ed-9a7e-edc36297c59a\") "
Oct 30 22:41:35.939654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939537    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:35.939863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939827    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.940007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.939985    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.940081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.940014    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.940081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.940032    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.940081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.940048    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.940081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.940063    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.941389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.941354    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.941494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.941368    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "kube-api-access-tq765". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.941566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.941526    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:35.941676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:35.941653    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy" pod "960c1ab3-72c2-45ed-9a7e-edc36297c59a" (UID: "960c1ab3-72c2-45ed-9a7e-edc36297c59a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:36.040826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.040798    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.040826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.040824    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq765\" (UniqueName: \"kubernetes.io/projected/960c1ab3-72c2-45ed-9a7e-edc36297c59a-kube-api-access-tq765\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.041019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.040834    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.041019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.040843    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c1ab3-72c2-45ed-9a7e-edc36297c59a-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.807924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.807892    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"
Oct 30 22:41:36.807924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.807903    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5" event={"ID":"960c1ab3-72c2-45ed-9a7e-edc36297c59a","Type":"ContainerDied","Data":"6fcfd40008892461a0a18e30a8a8efb51c7308d55ac1b252a74e3089fe28f6b9"}
Oct 30 22:41:36.808445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.807969    4410 scope.go:122] "RemoveContainer" containerID="0ac224a10cff11f7cbcc99c8bb499a964325f271ec1f07265869826c46df1810"
Oct 30 22:41:36.972400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:36.972368    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:41:37.021744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:37.021713    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-bpgb5"]
Oct 30 22:41:39.019471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:39.019436    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" path="/var/lib/kubelet/pods/960c1ab3-72c2-45ed-9a7e-edc36297c59a/volumes"
Oct 30 22:41:39.442588 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:39.442556    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:42.308767 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.308733    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-9cf96a"]
Oct 30 22:41:42.309187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309172    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c3a8d7df-1138-41cc-a9c5-59182d6aac93" containerName="revoke-keys"
Oct 30 22:41:42.309237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309187    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a8d7df-1138-41cc-a9c5-59182d6aac93" containerName="revoke-keys"
Oct 30 22:41:42.309237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309205    4410 cpu_manager.go:414] "containerMap: removing container" podUID="258c7c37-fefd-4260-b730-cf12816855e2" containerName="main"
Oct 30 22:41:42.309237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309210    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c7c37-fefd-4260-b730-cf12816855e2" containerName="main"
Oct 30 22:41:42.309336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309241    4410 cpu_manager.go:414] "containerMap: removing container" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy"
Oct 30 22:41:42.309336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309246    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy"
Oct 30 22:41:42.309336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309316    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a8d7df-1138-41cc-a9c5-59182d6aac93" containerName="revoke-keys"
Oct 30 22:41:42.309336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309325    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c7c37-fefd-4260-b730-cf12816855e2" containerName="main"
Oct 30 22:41:42.309336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.309334    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="960c1ab3-72c2-45ed-9a7e-edc36297c59a" containerName="istio-proxy"
Oct 30 22:41:42.313829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.313810    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:42.367247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.367226    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbdpx\""
Oct 30 22:41:42.369329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.369302    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-9cf96a"]
Oct 30 22:41:42.407815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.407769    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7zk\" (UniqueName: \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\") pod \"test-curl-7370-9cf96a\" (UID: \"19f2e6ca-513d-46e6-a109-8072747b10dd\") " pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:42.509319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.509286    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7zk\" (UniqueName: \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\") pod \"test-curl-7370-9cf96a\" (UID: \"19f2e6ca-513d-46e6-a109-8072747b10dd\") " pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:42.569407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.569340    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7zk\" (UniqueName: \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\") pod \"test-curl-7370-9cf96a\" (UID: \"19f2e6ca-513d-46e6-a109-8072747b10dd\") " pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:42.625023 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.624995    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:42.996420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:42.996393    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-9cf96a"]
Oct 30 22:41:42.998477 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:41:42.998450    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f2e6ca_513d_46e6_a109_8072747b10dd.slice/crio-56960dd6eeae4fb42e50bee85c55e1dfa701172507290bca62d170de2c0886bd WatchSource:0}: Error finding container 56960dd6eeae4fb42e50bee85c55e1dfa701172507290bca62d170de2c0886bd: Status 404 returned error can't find the container with id 56960dd6eeae4fb42e50bee85c55e1dfa701172507290bca62d170de2c0886bd
Oct 30 22:41:43.858843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:43.858808    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-9cf96a" event={"ID":"19f2e6ca-513d-46e6-a109-8072747b10dd","Type":"ContainerStarted","Data":"c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7"}
Oct 30 22:41:43.859221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:43.858855    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-9cf96a" event={"ID":"19f2e6ca-513d-46e6-a109-8072747b10dd","Type":"ContainerStarted","Data":"56960dd6eeae4fb42e50bee85c55e1dfa701172507290bca62d170de2c0886bd"}
Oct 30 22:41:43.973255 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:43.973198    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-9cf96a" podStartSLOduration=1.973184339 podStartE2EDuration="1.973184339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:43.971053469 +0000 UTC m=+3557.059791274" watchObservedRunningTime="2026-10-30 22:41:43.973184339 +0000 UTC m=+3557.061922141"
Oct 30 22:41:44.398295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:44.398260    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:49.221509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:49.221476    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:50.522610 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:41:50.522575    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:41:54.042262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:54.042229    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:56.791073 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.791028    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"]
Oct 30 22:41:56.794972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.794952    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.847648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.847623    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:56.853173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.853146    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.853426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.853394    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.853546 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.853516    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7k86\" (UniqueName: \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.896183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.896155    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:56.896183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.896171    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:56.897065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.897037    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:41:56.899580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.899557    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"]
Oct 30 22:41:56.955247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.955218    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.955420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.955285    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7k86\" (UniqueName: \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.955499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.955443    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.956015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.955992    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:56.957738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:56.957710    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:57.011909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.011877    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7k86\" (UniqueName: \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\") pod \"maas-api-revoke-sub-035aa228-r6lwc\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:57.108493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.108430    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:41:57.479395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.479367    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"]
Oct 30 22:41:57.481446 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:41:57.481420    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode405797c_afab_4a20_8a38_a7642141ce6e.slice/crio-ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b WatchSource:0}: Error finding container ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b: Status 404 returned error can't find the container with id ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b
Oct 30 22:41:57.826795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.826734    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:57.936260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.936226    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc" event={"ID":"e405797c-afab-4a20-8a38-a7642141ce6e","Type":"ContainerStarted","Data":"2384adaccbffb4ae15821de989197cf0e872e79fdce48117f0fc6bbefa42c1f7"}
Oct 30 22:41:57.936399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:57.936287    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc" event={"ID":"e405797c-afab-4a20-8a38-a7642141ce6e","Type":"ContainerStarted","Data":"ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b"}
Oct 30 22:41:58.066047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.065953    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc" podStartSLOduration=2.065940635 podStartE2EDuration="2.065940635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:58.063815175 +0000 UTC m=+3571.152552979" watchObservedRunningTime="2026-10-30 22:41:58.065940635 +0000 UTC m=+3571.154678438"
Oct 30 22:41:58.412926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.412892    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.493306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.493271    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7376-3a9827"]
Oct 30 22:41:58.496896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.496880    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:41:58.553336 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.553311    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-3a9827"]
Oct 30 22:41:58.573233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.573208    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bsp\" (UniqueName: \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\") pod \"test-curl-7376-3a9827\" (UID: \"c1ff5e42-64ea-4b42-baea-5d7c44219737\") " pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:41:58.674113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.674084    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bsp\" (UniqueName: \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\") pod \"test-curl-7376-3a9827\" (UID: \"c1ff5e42-64ea-4b42-baea-5d7c44219737\") " pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:41:58.730540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.730511    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bsp\" (UniqueName: \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\") pod \"test-curl-7376-3a9827\" (UID: \"c1ff5e42-64ea-4b42-baea-5d7c44219737\") " pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:41:58.809600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.809570    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:41:58.900458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.900426    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.953208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.953122    4410 generic.go:356] "Generic (PLEG): container finished" podID="e405797c-afab-4a20-8a38-a7642141ce6e" containerID="2384adaccbffb4ae15821de989197cf0e872e79fdce48117f0fc6bbefa42c1f7" exitCode=0
Oct 30 22:41:58.953364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:58.953199    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc" event={"ID":"e405797c-afab-4a20-8a38-a7642141ce6e","Type":"ContainerDied","Data":"2384adaccbffb4ae15821de989197cf0e872e79fdce48117f0fc6bbefa42c1f7"}
Oct 30 22:41:59.133407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.133371    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-9cf96a"]
Oct 30 22:41:59.134409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.133603    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-9cf96a" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" containerName="test-curl-7370-9cf96a" containerID="cri-o://c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7" gracePeriod=2
Oct 30 22:41:59.136542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.136517    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-9cf96a"]
Oct 30 22:41:59.180216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.180190    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7376-3a9827"]
Oct 30 22:41:59.182336 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:41:59.182312    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ff5e42_64ea_4b42_baea_5d7c44219737.slice/crio-6854be717d2d44b4b629b479b8c27ff7696af0f188ccce5a3611583a3eba1566 WatchSource:0}: Error finding container 6854be717d2d44b4b629b479b8c27ff7696af0f188ccce5a3611583a3eba1566: Status 404 returned error can't find the container with id 6854be717d2d44b4b629b479b8c27ff7696af0f188ccce5a3611583a3eba1566
Oct 30 22:41:59.184167 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:41:59.184129    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:59.292873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.292847    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-a60adf"]
Oct 30 22:41:59.293308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.293295    4410 cpu_manager.go:414] "containerMap: removing container" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" containerName="test-curl-7370-9cf96a"
Oct 30 22:41:59.293361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.293309    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" containerName="test-curl-7370-9cf96a"
Oct 30 22:41:59.293398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.293387    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" containerName="test-curl-7370-9cf96a"
Oct 30 22:41:59.296551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.296536    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:41:59.347109 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:41:59.347075    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:41:59.354025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.353999    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-a60adf"]
Oct 30 22:41:59.381936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.381906    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cz4v\" (UniqueName: \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\") pod \"test-curl-7370-a60adf\" (UID: \"a23e4326-a442-4ace-b480-097667ab5cf8\") " pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:41:59.483513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.483436    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cz4v\" (UniqueName: \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\") pod \"test-curl-7370-a60adf\" (UID: \"a23e4326-a442-4ace-b480-097667ab5cf8\") " pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:41:59.539729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.539705    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cz4v\" (UniqueName: \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\") pod \"test-curl-7370-a60adf\" (UID: \"a23e4326-a442-4ace-b480-097667ab5cf8\") " pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:41:59.608594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.608566    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:41:59.960485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.960441    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-3a9827" event={"ID":"c1ff5e42-64ea-4b42-baea-5d7c44219737","Type":"ContainerStarted","Data":"2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1"}
Oct 30 22:41:59.960979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.960492    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7376-3a9827" event={"ID":"c1ff5e42-64ea-4b42-baea-5d7c44219737","Type":"ContainerStarted","Data":"6854be717d2d44b4b629b479b8c27ff7696af0f188ccce5a3611583a3eba1566"}
Oct 30 22:41:59.991559 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:41:59.991531    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-a60adf"]
Oct 30 22:41:59.994379 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:41:59.994353    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23e4326_a442_4ace_b480_097667ab5cf8.slice/crio-3b7fd4e6325fd13f3f21aea1a846aa8140b7b044f9f690ef8c851131c2152176 WatchSource:0}: Error finding container 3b7fd4e6325fd13f3f21aea1a846aa8140b7b044f9f690ef8c851131c2152176: Status 404 returned error can't find the container with id 3b7fd4e6325fd13f3f21aea1a846aa8140b7b044f9f690ef8c851131c2152176
Oct 30 22:42:00.010966 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:00.010934    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:00.092040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.092018    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:42:00.124501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.124444    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7376-3a9827" podStartSLOduration=2.124427483 podStartE2EDuration="2.124427483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:00.123342404 +0000 UTC m=+3573.212080206" watchObservedRunningTime="2026-10-30 22:42:00.124427483 +0000 UTC m=+3573.213165290"
Oct 30 22:42:00.173750 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:00.173720    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:00.190166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.190062    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\" (UniqueName: \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\") pod \"e405797c-afab-4a20-8a38-a7642141ce6e\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") "
Oct 30 22:42:00.190166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.190154    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\") pod \"e405797c-afab-4a20-8a38-a7642141ce6e\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") "
Oct 30 22:42:00.190361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.190207    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\") pod \"e405797c-afab-4a20-8a38-a7642141ce6e\" (UID: \"e405797c-afab-4a20-8a38-a7642141ce6e\") "
Oct 30 22:42:00.190548 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.190520    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca" pod "e405797c-afab-4a20-8a38-a7642141ce6e" (UID: "e405797c-afab-4a20-8a38-a7642141ce6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:00.190722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.190699    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:00.192358 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.192336    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca" pod "e405797c-afab-4a20-8a38-a7642141ce6e" (UID: "e405797c-afab-4a20-8a38-a7642141ce6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:00.192514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.192469    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86" pod "e405797c-afab-4a20-8a38-a7642141ce6e" (UID: "e405797c-afab-4a20-8a38-a7642141ce6e"). InnerVolumeSpecName "kube-api-access-b7k86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:00.292363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.292295    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e405797c-afab-4a20-8a38-a7642141ce6e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:00.292363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.292324    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7k86\" (UniqueName: \"kubernetes.io/projected/e405797c-afab-4a20-8a38-a7642141ce6e-kube-api-access-b7k86\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:00.967528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.967486    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-a60adf" event={"ID":"a23e4326-a442-4ace-b480-097667ab5cf8","Type":"ContainerStarted","Data":"f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501"}
Oct 30 22:42:00.968037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.967540    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-a60adf" event={"ID":"a23e4326-a442-4ace-b480-097667ab5cf8","Type":"ContainerStarted","Data":"3b7fd4e6325fd13f3f21aea1a846aa8140b7b044f9f690ef8c851131c2152176"}
Oct 30 22:42:00.969041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.969013    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc" event={"ID":"e405797c-afab-4a20-8a38-a7642141ce6e","Type":"ContainerDied","Data":"ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b"}
Oct 30 22:42:00.969144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.969069    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc26e0ff09911d6dba48d68d78f3a69b187d33ce810548429a8728741b3984b"
Oct 30 22:42:00.969144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:00.969028    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"
Oct 30 22:42:01.018220 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:01.018184    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:01.131286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.131235    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-a60adf" podStartSLOduration=2.131222043 podStartE2EDuration="2.131222043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:01.129133812 +0000 UTC m=+3574.217871616" watchObservedRunningTime="2026-10-30 22:42:01.131222043 +0000 UTC m=+3574.219959845"
Oct 30 22:42:01.383018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.382991    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:01.404367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.404343    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\" (UniqueName: \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\") pod \"19f2e6ca-513d-46e6-a109-8072747b10dd\" (UID: \"19f2e6ca-513d-46e6-a109-8072747b10dd\") "
Oct 30 22:42:01.406264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.406239    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk" pod "19f2e6ca-513d-46e6-a109-8072747b10dd" (UID: "19f2e6ca-513d-46e6-a109-8072747b10dd"). InnerVolumeSpecName "kube-api-access-nb7zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:01.505477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.505447    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7zk\" (UniqueName: \"kubernetes.io/projected/19f2e6ca-513d-46e6-a109-8072747b10dd-kube-api-access-nb7zk\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:01.543471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.543429    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7376-3a9827"]
Oct 30 22:42:01.550904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.550875    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7376-3a9827"]
Oct 30 22:42:01.976193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.976158    4410 generic.go:356] "Generic (PLEG): container finished" podID="19f2e6ca-513d-46e6-a109-8072747b10dd" containerID="c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7" exitCode=137
Oct 30 22:42:01.976677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.976228    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:01.976677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.976248    4410 scope.go:122] "RemoveContainer" containerID="c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7"
Oct 30 22:42:01.976677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.976537    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7376-3a9827" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" containerName="test-curl-7376-3a9827" containerID="cri-o://2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1" gracePeriod=2
Oct 30 22:42:01.985216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.985200    4410 scope.go:122] "RemoveContainer" containerID="c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7"
Oct 30 22:42:01.985459 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:01.985441    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7\": container with ID starting with c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7 not found: ID does not exist" containerID="c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7"
Oct 30 22:42:01.985504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:01.985467    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7"} err="failed to get container status \"c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7\": rpc error: code = NotFound desc = could not find container \"c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7\": container with ID starting with c44ee33c64cefc12ee0e91f717e341d00a8fc90efc05fdd2e411deedd799cdc7 not found: ID does not exist"
Oct 30 22:42:02.027018 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:02.026980    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-3a9827\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:02.077511 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:02.077484    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:02.128081 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:02.128051    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-3a9827\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:02.515306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.515273    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-a60adf"]
Oct 30 22:42:02.530140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.530105    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-a60adf"]
Oct 30 22:42:02.677260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677229    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-72f4fb"]
Oct 30 22:42:02.677675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677662    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" containerName="test-curl-7376-3a9827"
Oct 30 22:42:02.677725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677676    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" containerName="test-curl-7376-3a9827"
Oct 30 22:42:02.677725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677687    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e405797c-afab-4a20-8a38-a7642141ce6e" containerName="revoke-keys"
Oct 30 22:42:02.677725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677693    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e405797c-afab-4a20-8a38-a7642141ce6e" containerName="revoke-keys"
Oct 30 22:42:02.677725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677705    4410 cpu_manager.go:414] "containerMap: removing container" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" containerName="test-curl-7370-a60adf"
Oct 30 22:42:02.677725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677710    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" containerName="test-curl-7370-a60adf"
Oct 30 22:42:02.677916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677824    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" containerName="test-curl-7376-3a9827"
Oct 30 22:42:02.677916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677834    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e405797c-afab-4a20-8a38-a7642141ce6e" containerName="revoke-keys"
Oct 30 22:42:02.677916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.677841    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" containerName="test-curl-7370-a60adf"
Oct 30 22:42:02.680921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.680905    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:02.719636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.719612    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8dd\" (UniqueName: \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\") pod \"test-curl-7370-72f4fb\" (UID: \"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa\") " pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:02.736432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.736406    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-72f4fb"]
Oct 30 22:42:02.820543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.820467    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8dd\" (UniqueName: \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\") pod \"test-curl-7370-72f4fb\" (UID: \"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa\") " pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:02.844716 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:02.844682    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:02.879027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.879005    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8dd\" (UniqueName: \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\") pod \"test-curl-7370-72f4fb\" (UID: \"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa\") " pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:02.895175 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:02.895140    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-3a9827\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:02.983488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.983454    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-a60adf" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" containerName="test-curl-7370-a60adf" containerID="cri-o://f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501" gracePeriod=2
Oct 30 22:42:02.992223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:02.992201    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:03.027567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.021856    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" path="/var/lib/kubelet/pods/19f2e6ca-513d-46e6-a109-8072747b10dd/volumes"
Oct 30 22:42:03.034016 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:03.033984    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-9cf96a\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="19f2e6ca-513d-46e6-a109-8072747b10dd" pod="opendatahub/test-curl-7370-9cf96a"
Oct 30 22:42:03.084494 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:03.084414    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7376-3a9827\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:03.134687 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:03.134653    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-a60adf\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:42:03.330432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.330396    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:03.370549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.370522    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-72f4fb"]
Oct 30 22:42:03.372217 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:03.372191    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4973e4f_3f6d_4beb_81b7_b516f8ced3fa.slice/crio-666ce0ce812999bf30acbc8ada317370f7dccb37c997976a2eaf527165fe56a8 WatchSource:0}: Error finding container 666ce0ce812999bf30acbc8ada317370f7dccb37c997976a2eaf527165fe56a8: Status 404 returned error can't find the container with id 666ce0ce812999bf30acbc8ada317370f7dccb37c997976a2eaf527165fe56a8
Oct 30 22:42:03.758132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.758093    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:03.992715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.992676    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-72f4fb" event={"ID":"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa","Type":"ContainerStarted","Data":"1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495"}
Oct 30 22:42:03.993161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:03.992729    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-72f4fb" event={"ID":"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa","Type":"ContainerStarted","Data":"666ce0ce812999bf30acbc8ada317370f7dccb37c997976a2eaf527165fe56a8"}
Oct 30 22:42:04.049470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.049400    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:04.106229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.106166    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-72f4fb" podStartSLOduration=2.10615 podStartE2EDuration="2.10615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:04.10456808 +0000 UTC m=+3577.193305884" watchObservedRunningTime="2026-10-30 22:42:04.10615 +0000 UTC m=+3577.194887803"
Oct 30 22:42:04.220459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.220435    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:04.337469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.337381    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\" (UniqueName: \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\") pod \"c1ff5e42-64ea-4b42-baea-5d7c44219737\" (UID: \"c1ff5e42-64ea-4b42-baea-5d7c44219737\") "
Oct 30 22:42:04.339568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.339537    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp" pod "c1ff5e42-64ea-4b42-baea-5d7c44219737" (UID: "c1ff5e42-64ea-4b42-baea-5d7c44219737"). InnerVolumeSpecName "kube-api-access-f7bsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:04.405076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.405040    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:04.439080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:04.439050    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bsp\" (UniqueName: \"kubernetes.io/projected/c1ff5e42-64ea-4b42-baea-5d7c44219737-kube-api-access-f7bsp\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:05.000806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.000758    4410 generic.go:356] "Generic (PLEG): container finished" podID="c1ff5e42-64ea-4b42-baea-5d7c44219737" containerID="2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1" exitCode=137
Oct 30 22:42:05.001275 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.000824    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7376-3a9827"
Oct 30 22:42:05.001275 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.000851    4410 scope.go:122] "RemoveContainer" containerID="2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1"
Oct 30 22:42:05.019320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.019285    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ff5e42-64ea-4b42-baea-5d7c44219737" path="/var/lib/kubelet/pods/c1ff5e42-64ea-4b42-baea-5d7c44219737/volumes"
Oct 30 22:42:05.045227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.045206    4410 scope.go:122] "RemoveContainer" containerID="2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1"
Oct 30 22:42:05.045544 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:05.045519    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1\": container with ID starting with 2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1 not found: ID does not exist" containerID="2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1"
Oct 30 22:42:05.045641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.045552    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1"} err="failed to get container status \"2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1\": rpc error: code = NotFound desc = could not find container \"2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1\": container with ID starting with 2bdaabd2862eed2a74142a63f26f83b50cd9016e173deb5f72fea661a3eeafa1 not found: ID does not exist"
Oct 30 22:42:05.232982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.232956    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:42:05.349107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.349028    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\" (UniqueName: \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\") pod \"a23e4326-a442-4ace-b480-097667ab5cf8\" (UID: \"a23e4326-a442-4ace-b480-097667ab5cf8\") "
Oct 30 22:42:05.351328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.351293    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v" pod "a23e4326-a442-4ace-b480-097667ab5cf8" (UID: "a23e4326-a442-4ace-b480-097667ab5cf8"). InnerVolumeSpecName "kube-api-access-4cz4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:05.450753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.450721    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cz4v\" (UniqueName: \"kubernetes.io/projected/a23e4326-a442-4ace-b480-097667ab5cf8-kube-api-access-4cz4v\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:05.628165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.628089    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-72f4fb"]
Oct 30 22:42:05.630766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.630738    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-72f4fb"]
Oct 30 22:42:05.770619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.770591    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-c9b104"]
Oct 30 22:42:05.771082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.771067    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" containerName="test-curl-7370-72f4fb"
Oct 30 22:42:05.771148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.771084    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" containerName="test-curl-7370-72f4fb"
Oct 30 22:42:05.771193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.771158    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" containerName="test-curl-7370-72f4fb"
Oct 30 22:42:05.774323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.774308    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:05.834229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.833278    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-c9b104"]
Oct 30 22:42:05.854543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.854508    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s94g\" (UniqueName: \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\") pod \"test-curl-7370-c9b104\" (UID: \"bffd1037-972d-4241-94ea-ed369d9421e2\") " pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:05.955810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:05.955755    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s94g\" (UniqueName: \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\") pod \"test-curl-7370-c9b104\" (UID: \"bffd1037-972d-4241-94ea-ed369d9421e2\") " pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:06.010371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.010334    4410 generic.go:356] "Generic (PLEG): container finished" podID="a23e4326-a442-4ace-b480-097667ab5cf8" containerID="f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501" exitCode=137
Oct 30 22:42:06.010862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.010382    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:42:06.010862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.010421    4410 scope.go:122] "RemoveContainer" containerID="f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501"
Oct 30 22:42:06.010862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.010641    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-72f4fb" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" containerName="test-curl-7370-72f4fb" containerID="cri-o://1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495" gracePeriod=2
Oct 30 22:42:06.014887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.014860    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s94g\" (UniqueName: \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\") pod \"test-curl-7370-c9b104\" (UID: \"bffd1037-972d-4241-94ea-ed369d9421e2\") " pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:06.030720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.030702    4410 scope.go:122] "RemoveContainer" containerID="f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501"
Oct 30 22:42:06.031053 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:06.031029    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501\": container with ID starting with f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501 not found: ID does not exist" containerID="f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501"
Oct 30 22:42:06.031143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.031069    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501"} err="failed to get container status \"f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501\": rpc error: code = NotFound desc = could not find container \"f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501\": container with ID starting with f0494cab997c39071d2a6175f180880a50afa24e16c48696996bd17115fcd501 not found: ID does not exist"
Oct 30 22:42:06.061132 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:06.061100    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-72f4fb\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:06.085832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.085809    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:06.111973 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:06.111936    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-a60adf\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:42:06.162725 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:06.162679    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-72f4fb\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:06.462741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:06.462714    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-c9b104"]
Oct 30 22:42:06.464409 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:06.464379    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbffd1037_972d_4241_94ea_ed369d9421e2.slice/crio-8921fa5b59d4b509b9e8d3012ff3a9bb4eb031c3526d5a2d17b0b5b47b141ba9 WatchSource:0}: Error finding container 8921fa5b59d4b509b9e8d3012ff3a9bb4eb031c3526d5a2d17b0b5b47b141ba9: Status 404 returned error can't find the container with id 8921fa5b59d4b509b9e8d3012ff3a9bb4eb031c3526d5a2d17b0b5b47b141ba9
Oct 30 22:42:07.019942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.019907    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" path="/var/lib/kubelet/pods/a23e4326-a442-4ace-b480-097667ab5cf8/volumes"
Oct 30 22:42:07.020301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.020279    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-c9b104" event={"ID":"bffd1037-972d-4241-94ea-ed369d9421e2","Type":"ContainerStarted","Data":"374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84"}
Oct 30 22:42:07.020363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.020313    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-c9b104" event={"ID":"bffd1037-972d-4241-94ea-ed369d9421e2","Type":"ContainerStarted","Data":"8921fa5b59d4b509b9e8d3012ff3a9bb4eb031c3526d5a2d17b0b5b47b141ba9"}
Oct 30 22:42:07.066508 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:07.066455    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-72f4fb\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:07.117324 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:07.117284    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-a60adf\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="a23e4326-a442-4ace-b480-097667ab5cf8" pod="opendatahub/test-curl-7370-a60adf"
Oct 30 22:42:07.232438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.232387    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-c9b104" podStartSLOduration=2.232374953 podStartE2EDuration="2.232374953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:07.231177787 +0000 UTC m=+3580.319915591" watchObservedRunningTime="2026-10-30 22:42:07.232374953 +0000 UTC m=+3580.321112755"
Oct 30 22:42:07.539532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.539502    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"]
Oct 30 22:42:07.543281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.543264    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.595416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.595390    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:07.600078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.600060    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:07.644843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.644812    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:07.651923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.651891    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"]
Oct 30 22:42:07.676158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.676131    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.676248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.676162    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.676248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.676185    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjnj\" (UniqueName: \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.695643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.695621    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:42:07.777153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.777126    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.777267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.777157    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.777267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.777185    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjnj\" (UniqueName: \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.777727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.777708    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.779403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.779386    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.834010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.833957    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjnj\" (UniqueName: \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\") pod \"maas-api-revoke-sub-29b56146-k4jqz\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:07.856603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:07.856582    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:08.231314 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.231283    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"]
Oct 30 22:42:08.232217 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:08.232188    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b129ce8_5517_4d01_b125_cf155d1bdd1d.slice/crio-9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa WatchSource:0}: Error finding container 9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa: Status 404 returned error can't find the container with id 9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa
Oct 30 22:42:08.241586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.241557    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:08.382954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.382905    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\" (UniqueName: \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\") pod \"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa\" (UID: \"b4973e4f-3f6d-4beb-81b7-b516f8ced3fa\") "
Oct 30 22:42:08.385816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.385756    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd" pod "b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" (UID: "b4973e4f-3f6d-4beb-81b7-b516f8ced3fa"). InnerVolumeSpecName "kube-api-access-xg8dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:08.484169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.484144    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8dd\" (UniqueName: \"kubernetes.io/projected/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa-kube-api-access-xg8dd\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:08.585067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.585036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:08.619923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.619892    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-c9b104"]
Oct 30 22:42:08.623970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.623939    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-c9b104"]
Oct 30 22:42:08.775093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.775007    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-628894"]
Oct 30 22:42:08.775481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.775465    4410 cpu_manager.go:414] "containerMap: removing container" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" containerName="test-curl-7370-c9b104"
Oct 30 22:42:08.775584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.775479    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" containerName="test-curl-7370-c9b104"
Oct 30 22:42:08.775584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.775579    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" containerName="test-curl-7370-c9b104"
Oct 30 22:42:08.778804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.778763    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:08.836808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.836764    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-628894"]
Oct 30 22:42:08.888155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.888122    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhdz\" (UniqueName: \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\") pod \"test-curl-7370-628894\" (UID: \"009bb095-f289-473a-a6c6-3de9f41db981\") " pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:08.988903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:08.988868    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhdz\" (UniqueName: \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\") pod \"test-curl-7370-628894\" (UID: \"009bb095-f289-473a-a6c6-3de9f41db981\") " pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:09.018969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.018932    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" path="/var/lib/kubelet/pods/b4973e4f-3f6d-4beb-81b7-b516f8ced3fa/volumes"
Oct 30 22:42:09.031657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.031628    4410 generic.go:356] "Generic (PLEG): container finished" podID="b4973e4f-3f6d-4beb-81b7-b516f8ced3fa" containerID="1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495" exitCode=137
Oct 30 22:42:09.031810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.031679    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-72f4fb"
Oct 30 22:42:09.031810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.031728    4410 scope.go:122] "RemoveContainer" containerID="1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495"
Oct 30 22:42:09.033802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.033672    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz" event={"ID":"4b129ce8-5517-4d01-b125-cf155d1bdd1d","Type":"ContainerStarted","Data":"3c39e94d8efcfa9e4a9de4cd0071eba502d6f3fcb8f8f237cc8480f8a3fb9ccc"}
Oct 30 22:42:09.033802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.033735    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz" event={"ID":"4b129ce8-5517-4d01-b125-cf155d1bdd1d","Type":"ContainerStarted","Data":"9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa"}
Oct 30 22:42:09.033984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.033837    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-c9b104" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" containerName="test-curl-7370-c9b104" containerID="cri-o://374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84" gracePeriod=2
Oct 30 22:42:09.041623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.041602    4410 scope.go:122] "RemoveContainer" containerID="1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495"
Oct 30 22:42:09.041897 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:09.041878    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495\": container with ID starting with 1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495 not found: ID does not exist" containerID="1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495"
Oct 30 22:42:09.041970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.041908    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495"} err="failed to get container status \"1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495\": rpc error: code = NotFound desc = could not find container \"1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495\": container with ID starting with 1fd310e2c471bb5e47a000d25c8205ff1bf7b9c103e233c09c83f71d1b8cd495 not found: ID does not exist"
Oct 30 22:42:09.045300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.045278    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhdz\" (UniqueName: \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\") pod \"test-curl-7370-628894\" (UID: \"009bb095-f289-473a-a6c6-3de9f41db981\") " pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:09.105282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.105257    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:09.148652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.148603    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz" podStartSLOduration=2.14858909 podStartE2EDuration="2.14858909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:09.14624269 +0000 UTC m=+3582.234980494" watchObservedRunningTime="2026-10-30 22:42:09.14858909 +0000 UTC m=+3582.237326893"
Oct 30 22:42:09.497151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:09.497123    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-628894"]
Oct 30 22:42:09.499157 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:09.499127    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod009bb095_f289_473a_a6c6_3de9f41db981.slice/crio-9c75becdcbc3199650f69ab61e1dae7f68fc41ed47787338a768a5b184c930f7 WatchSource:0}: Error finding container 9c75becdcbc3199650f69ab61e1dae7f68fc41ed47787338a768a5b184c930f7: Status 404 returned error can't find the container with id 9c75becdcbc3199650f69ab61e1dae7f68fc41ed47787338a768a5b184c930f7
Oct 30 22:42:10.041702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.041670    4410 generic.go:356] "Generic (PLEG): container finished" podID="4b129ce8-5517-4d01-b125-cf155d1bdd1d" containerID="3c39e94d8efcfa9e4a9de4cd0071eba502d6f3fcb8f8f237cc8480f8a3fb9ccc" exitCode=0
Oct 30 22:42:10.041885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.041755    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz" event={"ID":"4b129ce8-5517-4d01-b125-cf155d1bdd1d","Type":"ContainerDied","Data":"3c39e94d8efcfa9e4a9de4cd0071eba502d6f3fcb8f8f237cc8480f8a3fb9ccc"}
Oct 30 22:42:10.043370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.043339    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-628894" event={"ID":"009bb095-f289-473a-a6c6-3de9f41db981","Type":"ContainerStarted","Data":"d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720"}
Oct 30 22:42:10.043515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.043382    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-628894" event={"ID":"009bb095-f289-473a-a6c6-3de9f41db981","Type":"ContainerStarted","Data":"9c75becdcbc3199650f69ab61e1dae7f68fc41ed47787338a768a5b184c930f7"}
Oct 30 22:42:10.268342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.268281    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-628894" podStartSLOduration=2.268264184 podStartE2EDuration="2.268264184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:10.267305988 +0000 UTC m=+3583.356043793" watchObservedRunningTime="2026-10-30 22:42:10.268264184 +0000 UTC m=+3583.357001989"
Oct 30 22:42:10.916350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.916320    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:42:10.934511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:10.934479    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:11.221880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.221855    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:11.279865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.279840    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:11.319175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.319143    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\") pod \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") "
Oct 30 22:42:11.319331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.319270    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\") pod \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") "
Oct 30 22:42:11.319331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.319308    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\" (UniqueName: \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\") pod \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\" (UID: \"4b129ce8-5517-4d01-b125-cf155d1bdd1d\") "
Oct 30 22:42:11.319496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.319473    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca" pod "4b129ce8-5517-4d01-b125-cf155d1bdd1d" (UID: "4b129ce8-5517-4d01-b125-cf155d1bdd1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:11.319706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.319692    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:11.321312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.321290    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca" pod "4b129ce8-5517-4d01-b125-cf155d1bdd1d" (UID: "4b129ce8-5517-4d01-b125-cf155d1bdd1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:11.321408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.321342    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj" pod "4b129ce8-5517-4d01-b125-cf155d1bdd1d" (UID: "4b129ce8-5517-4d01-b125-cf155d1bdd1d"). InnerVolumeSpecName "kube-api-access-7cjnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:11.420432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.420405    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\" (UniqueName: \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\") pod \"bffd1037-972d-4241-94ea-ed369d9421e2\" (UID: \"bffd1037-972d-4241-94ea-ed369d9421e2\") "
Oct 30 22:42:11.420751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.420736    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b129ce8-5517-4d01-b125-cf155d1bdd1d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:11.420862 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.420752    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjnj\" (UniqueName: \"kubernetes.io/projected/4b129ce8-5517-4d01-b125-cf155d1bdd1d-kube-api-access-7cjnj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:11.422289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.422265    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g" pod "bffd1037-972d-4241-94ea-ed369d9421e2" (UID: "bffd1037-972d-4241-94ea-ed369d9421e2"). InnerVolumeSpecName "kube-api-access-2s94g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:11.522221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.522142    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s94g\" (UniqueName: \"kubernetes.io/projected/bffd1037-972d-4241-94ea-ed369d9421e2-kube-api-access-2s94g\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:11.732134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.732100    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-628894"]
Oct 30 22:42:11.737913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:11.737881    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-628894"]
Oct 30 22:42:12.057370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.057340    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"
Oct 30 22:42:12.057370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.057378    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz" event={"ID":"4b129ce8-5517-4d01-b125-cf155d1bdd1d","Type":"ContainerDied","Data":"9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa"}
Oct 30 22:42:12.058063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.057619    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d5292753d37dce20be3d8cb623e1230fb402a63a91a93d6f407d821239af9fa"
Oct 30 22:42:12.058840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.058811    4410 generic.go:356] "Generic (PLEG): container finished" podID="bffd1037-972d-4241-94ea-ed369d9421e2" containerID="374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84" exitCode=137
Oct 30 22:42:12.058963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.058910    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:12.058963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.058907    4410 scope.go:122] "RemoveContainer" containerID="374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84"
Oct 30 22:42:12.059069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.058973    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-628894" podUID="009bb095-f289-473a-a6c6-3de9f41db981" containerName="test-curl-7370-628894" containerID="cri-o://d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720" gracePeriod=2
Oct 30 22:42:12.069538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.069515    4410 scope.go:122] "RemoveContainer" containerID="374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84"
Oct 30 22:42:12.069850 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:12.069820    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84\": container with ID starting with 374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84 not found: ID does not exist" containerID="374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84"
Oct 30 22:42:12.069971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:12.069860    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84"} err="failed to get container status \"374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84\": rpc error: code = NotFound desc = could not find container \"374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84\": container with ID starting with 374142926a91435693d4403e02eac2d8b01823d983b5f2b06ce73fb4c2e22d84 not found: ID does not exist"
Oct 30 22:42:12.109387 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:12.109354    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-628894\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="009bb095-f289-473a-a6c6-3de9f41db981" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:12.212556 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:12.212519    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-c9b104\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" pod="opendatahub/test-curl-7370-c9b104"
Oct 30 22:42:12.262905 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:12.262868    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-628894\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="009bb095-f289-473a-a6c6-3de9f41db981" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:13.019326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:13.019293    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffd1037-972d-4241-94ea-ed369d9421e2" path="/var/lib/kubelet/pods/bffd1037-972d-4241-94ea-ed369d9421e2/volumes"
Oct 30 22:42:13.414694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:13.414664    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:14.262724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:14.262696    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:14.350657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:14.350600    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\" (UniqueName: \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\") pod \"009bb095-f289-473a-a6c6-3de9f41db981\" (UID: \"009bb095-f289-473a-a6c6-3de9f41db981\") "
Oct 30 22:42:14.352634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:14.352608    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz" pod "009bb095-f289-473a-a6c6-3de9f41db981" (UID: "009bb095-f289-473a-a6c6-3de9f41db981"). InnerVolumeSpecName "kube-api-access-2bhdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:14.452118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:14.452087    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhdz\" (UniqueName: \"kubernetes.io/projected/009bb095-f289-473a-a6c6-3de9f41db981-kube-api-access-2bhdz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:15.019598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.019562    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009bb095-f289-473a-a6c6-3de9f41db981" path="/var/lib/kubelet/pods/009bb095-f289-473a-a6c6-3de9f41db981/volumes"
Oct 30 22:42:15.077763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.077729    4410 generic.go:356] "Generic (PLEG): container finished" podID="009bb095-f289-473a-a6c6-3de9f41db981" containerID="d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720" exitCode=137
Oct 30 22:42:15.077924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.077795    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-628894"
Oct 30 22:42:15.077924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.077828    4410 scope.go:122] "RemoveContainer" containerID="d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720"
Oct 30 22:42:15.086492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.086474    4410 scope.go:122] "RemoveContainer" containerID="d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720"
Oct 30 22:42:15.086735 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:15.086717    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720\": container with ID starting with d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720 not found: ID does not exist" containerID="d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720"
Oct 30 22:42:15.086807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:15.086743    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720"} err="failed to get container status \"d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720\": rpc error: code = NotFound desc = could not find container \"d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720\": container with ID starting with d83bfdd1c4e0033c225fe4a0a7402df65e96af5d9f5e0bd030944e728641e720 not found: ID does not exist"
Oct 30 22:42:17.230343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:17.230307    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:17.401511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:17.401476    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:18.448375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:18.448335    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:18.729274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:18.729197    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:18.888898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:18.888864    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:19.044675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:19.044591    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:20.521841 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:20.521800    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:42:21.947836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:21.947769    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.156729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.156692    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.496494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.496456    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:42:22.497269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497245    4410 cpu_manager.go:414] "containerMap: removing container" podUID="009bb095-f289-473a-a6c6-3de9f41db981" containerName="test-curl-7370-628894"
Oct 30 22:42:22.497269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497267    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009bb095-f289-473a-a6c6-3de9f41db981" containerName="test-curl-7370-628894"
Oct 30 22:42:22.497482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497302    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4b129ce8-5517-4d01-b125-cf155d1bdd1d" containerName="revoke-keys"
Oct 30 22:42:22.497482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497310    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b129ce8-5517-4d01-b125-cf155d1bdd1d" containerName="revoke-keys"
Oct 30 22:42:22.497482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497430    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b129ce8-5517-4d01-b125-cf155d1bdd1d" containerName="revoke-keys"
Oct 30 22:42:22.497482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.497449    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="009bb095-f289-473a-a6c6-3de9f41db981" containerName="test-curl-7370-628894"
Oct 30 22:42:22.502159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.502138    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.556732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.556699    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:42:22.602579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.602547    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a6900c-dockercfg-wtmnm\""
Oct 30 22:42:22.602740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.602613    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a6900c\""
Oct 30 22:42:22.634166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.634138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.634317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.634190    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvmg\" (UniqueName: \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.735898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.735856    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.736055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.735927    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvmg\" (UniqueName: \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.736535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.736513    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.792851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.792789    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvmg\" (UniqueName: \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\") pod \"payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:22.818421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:22.818401    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:23.192621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:23.192596    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:42:23.194605 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:23.194579    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd232b960_03da_47b7_a60c_89d02fa9117c.slice/crio-a6e4e8a62fe6a04a504348c463bc5499feffff5c38b185710e4ea4bce8783828 WatchSource:0}: Error finding container a6e4e8a62fe6a04a504348c463bc5499feffff5c38b185710e4ea4bce8783828: Status 404 returned error can't find the container with id a6e4e8a62fe6a04a504348c463bc5499feffff5c38b185710e4ea4bce8783828
Oct 30 22:42:23.287861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:23.287828    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:23.711748 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:23.711711    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:23.950712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:23.950668    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:24.127986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:24.127894    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" event={"ID":"d232b960-03da-47b7-a60c-89d02fa9117c","Type":"ContainerStarted","Data":"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"}
Oct 30 22:42:24.127986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:24.127957    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" event={"ID":"d232b960-03da-47b7-a60c-89d02fa9117c","Type":"ContainerStarted","Data":"a6e4e8a62fe6a04a504348c463bc5499feffff5c38b185710e4ea4bce8783828"}
Oct 30 22:42:24.128221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:24.128064    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:24.193337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:24.193304    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:24.254814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:24.246559    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" podStartSLOduration=2.24654105 podStartE2EDuration="2.24654105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:24.242833779 +0000 UTC m=+3597.331571582" watchObservedRunningTime="2026-10-30 22:42:24.24654105 +0000 UTC m=+3597.335278854"
Oct 30 22:42:25.939173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:25.939130    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" podUID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" containerName="main" containerID="cri-o://478d30cf500027a02054b305b809af3da16b4f28885d031c712aa0cc983d11fd" gracePeriod=45
Oct 30 22:42:26.144483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.144450    4410 generic.go:356] "Generic (PLEG): container finished" podID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" containerID="478d30cf500027a02054b305b809af3da16b4f28885d031c712aa0cc983d11fd" exitCode=0
Oct 30 22:42:26.144699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.144634    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" event={"ID":"09324fa8-a2a1-4eb7-ab38-8df1ecae4467","Type":"ContainerDied","Data":"478d30cf500027a02054b305b809af3da16b4f28885d031c712aa0cc983d11fd"}
Oct 30 22:42:26.204996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.204926    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:42:26.380644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380611    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.380852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380650    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.380852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380676    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.380852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380706    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\" (UniqueName: \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.380852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380728    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\" (UniqueName: \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.380852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380765    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\") pod \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\" (UID: \"09324fa8-a2a1-4eb7-ab38-8df1ecae4467\") "
Oct 30 22:42:26.381168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.380972    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:26.381168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.381094    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:26.381168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.381107    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:26.381339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.381319    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.381403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.381345    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.381403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.381362    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.382963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.382939    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:26.383080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.383020    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:26.383080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.383064    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d" pod "09324fa8-a2a1-4eb7-ab38-8df1ecae4467" (UID: "09324fa8-a2a1-4eb7-ab38-8df1ecae4467"). InnerVolumeSpecName "kube-api-access-zpx5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:26.482113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.482058    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.482113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.482081    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpx5d\" (UniqueName: \"kubernetes.io/projected/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-kube-api-access-zpx5d\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.482113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:26.482091    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09324fa8-a2a1-4eb7-ab38-8df1ecae4467-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:27.151710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:27.151684    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"
Oct 30 22:42:27.152214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:27.151730    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf" event={"ID":"09324fa8-a2a1-4eb7-ab38-8df1ecae4467","Type":"ContainerDied","Data":"b2f4bd508967ec8712a38ae9f386d9983fec5a4bc5bcc252612619a6d9558d96"}
Oct 30 22:42:27.152214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:27.151799    4410 scope.go:122] "RemoveContainer" containerID="478d30cf500027a02054b305b809af3da16b4f28885d031c712aa0cc983d11fd"
Oct 30 22:42:27.314970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:27.314894    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:42:27.317187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:27.317166    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6c788d/auth-test-model-6c788d-kserve-f4456f6b5-g6jmf"]
Oct 30 22:42:28.643727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:28.643696    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.847506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:28.847461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:29.016704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:29.016513    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:29.020551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:29.020522    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" path="/var/lib/kubelet/pods/09324fa8-a2a1-4eb7-ab38-8df1ecae4467/volumes"
Oct 30 22:42:31.950885 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:31.950849    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:32.128676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.128638    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:32.662139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.662109    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:42:32.662589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.662575    4410 cpu_manager.go:414] "containerMap: removing container" podUID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" containerName="main"
Oct 30 22:42:32.662642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.662591    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" containerName="main"
Oct 30 22:42:32.662704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.662695    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="09324fa8-a2a1-4eb7-ab38-8df1ecae4467" containerName="main"
Oct 30 22:42:32.667678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.667662    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:32.723931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.723899    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:42:32.769473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.769449    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-20ccef\""
Oct 30 22:42:32.769611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.769535    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-20ccef-dockercfg-6cggh\""
Oct 30 22:42:32.847872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.847842    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:32.848150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.848121    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxcm\" (UniqueName: \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:32.949469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.949409    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxcm\" (UniqueName: \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:32.949572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.949511    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:32.950156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:32.950134    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:33.006551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:33.006527    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxcm\" (UniqueName: \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\") pod \"payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:33.279883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:33.279801    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:33.658516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:33.658488    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:42:33.659702 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:33.659673    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c64a939_a39d_4b45_8924_b15c6ac5eb40.slice/crio-bb5a49ee13966f5cba33310aed9ad3114a47576346e3c4f3cf2145f35dd485b6 WatchSource:0}: Error finding container bb5a49ee13966f5cba33310aed9ad3114a47576346e3c4f3cf2145f35dd485b6: Status 404 returned error can't find the container with id bb5a49ee13966f5cba33310aed9ad3114a47576346e3c4f3cf2145f35dd485b6
Oct 30 22:42:33.784003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:33.783970    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:34.120881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.120799    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:34.192597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.192562    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" event={"ID":"6c64a939-a39d-4b45-8924-b15c6ac5eb40","Type":"ContainerStarted","Data":"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"}
Oct 30 22:42:34.192792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.192612    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" event={"ID":"6c64a939-a39d-4b45-8924-b15c6ac5eb40","Type":"ContainerStarted","Data":"bb5a49ee13966f5cba33310aed9ad3114a47576346e3c4f3cf2145f35dd485b6"}
Oct 30 22:42:34.192792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.192662    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:34.279026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.278989    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:34.311074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:34.311010    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" podStartSLOduration=2.310994106 podStartE2EDuration="2.310994106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:34.310381703 +0000 UTC m=+3607.399119507" watchObservedRunningTime="2026-10-30 22:42:34.310994106 +0000 UTC m=+3607.399731910"
Oct 30 22:42:35.138708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:35.138678    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:42:38.666120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:38.666087    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:43.614467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.614422    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"]
Oct 30 22:42:43.616736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.616710    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:43.618704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.618688    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.671232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.671204    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:43.674923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.674896    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.675074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.674947    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.675074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.675038    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.720869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.720839    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:43.721042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.720876    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:42:43.721042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.720941    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:43.723466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.723446    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"]
Oct 30 22:42:43.776218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.776182    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.776351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.776243    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.776351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.776281    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.776956 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.776933    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.778433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.778412    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.836015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.835992    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\") pod \"maas-api-revoke-sub-679796a9-77skn\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:43.933674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:43.933651    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:44.307039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:44.307008    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"]
Oct 30 22:42:44.308540 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:44.308507    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df2245e_8128_4b62_ad43_f6b802c36442.slice/crio-71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6 WatchSource:0}: Error finding container 71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6: Status 404 returned error can't find the container with id 71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6
Oct 30 22:42:45.200435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:45.200405    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:42:45.254566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:45.254537    4410 generic.go:356] "Generic (PLEG): container finished" podID="7df2245e-8128-4b62-ad43-f6b802c36442" containerID="353f384671c40e58bd65831fab6fb0a636c4c43f7097dba812affcc71505ce28" exitCode=0
Oct 30 22:42:45.254716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:45.254588    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn" event={"ID":"7df2245e-8128-4b62-ad43-f6b802c36442","Type":"ContainerDied","Data":"353f384671c40e58bd65831fab6fb0a636c4c43f7097dba812affcc71505ce28"}
Oct 30 22:42:45.254716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:45.254629    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn" event={"ID":"7df2245e-8128-4b62-ad43-f6b802c36442","Type":"ContainerStarted","Data":"71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6"}
Oct 30 22:42:46.281584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.281552    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:42:46.282021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.281740    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" containerID="cri-o://890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d" gracePeriod=30
Oct 30 22:42:46.404943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.404917    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:46.505352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.505320    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\") pod \"7df2245e-8128-4b62-ad43-f6b802c36442\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") "
Oct 30 22:42:46.505509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.505392    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\") pod \"7df2245e-8128-4b62-ad43-f6b802c36442\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") "
Oct 30 22:42:46.505509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.505492    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\") pod \"7df2245e-8128-4b62-ad43-f6b802c36442\" (UID: \"7df2245e-8128-4b62-ad43-f6b802c36442\") "
Oct 30 22:42:46.505797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.505746    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca" pod "7df2245e-8128-4b62-ad43-f6b802c36442" (UID: "7df2245e-8128-4b62-ad43-f6b802c36442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:46.505930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.505905    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:46.507555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.507532    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg" pod "7df2245e-8128-4b62-ad43-f6b802c36442" (UID: "7df2245e-8128-4b62-ad43-f6b802c36442"). InnerVolumeSpecName "kube-api-access-gbxrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:46.507914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.507894    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca" pod "7df2245e-8128-4b62-ad43-f6b802c36442" (UID: "7df2245e-8128-4b62-ad43-f6b802c36442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:46.607033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.607007    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/7df2245e-8128-4b62-ad43-f6b802c36442-kube-api-access-gbxrg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:46.607033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:46.607033    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df2245e-8128-4b62-ad43-f6b802c36442-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:47.266147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:47.266107    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn" event={"ID":"7df2245e-8128-4b62-ad43-f6b802c36442","Type":"ContainerDied","Data":"71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6"}
Oct 30 22:42:47.266147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:47.266153    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f62cf60493324a7be843a92665d14e3da33172269fa66f5409d9372dd77fc6"
Oct 30 22:42:47.266351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:47.266116    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"
Oct 30 22:42:48.469000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:48.468966    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:50.308301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:50.308266    4410 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:50.308708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:50.308317    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:50.521186 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:50.521149    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:42:51.543060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.543037    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:42:51.664091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664065    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664108    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664158    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664241    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664268    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664317    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664378    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664411    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664485    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\") pod \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\" (UID: \"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e\") "
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664487    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:51.664589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664557    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664651    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664938    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664954    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664973    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664984    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.665000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.664998    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.666529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.666493    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:51.666629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.666568    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:51.666744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.666728    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "kube-api-access-mmvlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:51.666805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.666728    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy" pod "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" (UID: "5fb225fa-2b45-4988-9d05-b83a1fb3ba2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:51.765747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.765718    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.765747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.765741    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvlj\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-kube-api-access-mmvlj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.765747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.765750    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.766029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.765760    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:51.766029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:51.765794    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:52.294519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.294482    4410 generic.go:356] "Generic (PLEG): container finished" podID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerID="890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d" exitCode=0
Oct 30 22:42:52.294687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.294528    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"
Oct 30 22:42:52.294687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.294574    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" event={"ID":"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e","Type":"ContainerDied","Data":"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"}
Oct 30 22:42:52.294687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.294621    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj" event={"ID":"5fb225fa-2b45-4988-9d05-b83a1fb3ba2e","Type":"ContainerDied","Data":"95b7bd42a5f0dfe14d3f505a46857f3de479195be3ee9c26a4be7629429f3fae"}
Oct 30 22:42:52.294687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.294642    4410 scope.go:122] "RemoveContainer" containerID="890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"
Oct 30 22:42:52.307634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.307610    4410 scope.go:122] "RemoveContainer" containerID="890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"
Oct 30 22:42:52.308079 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:42:52.308050    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d\": container with ID starting with 890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d not found: ID does not exist" containerID="890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"
Oct 30 22:42:52.308176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.308088    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d"} err="failed to get container status \"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d\": rpc error: code = NotFound desc = could not find container \"890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d\": container with ID starting with 890432f2e9d0323d3558a4ae15efc22378f83a0b957bc9481d36631b484ceb1d not found: ID does not exist"
Oct 30 22:42:52.508830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.508796    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:42:52.514021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:52.513997    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-9sczj"]
Oct 30 22:42:53.025995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:53.025964    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" path="/var/lib/kubelet/pods/5fb225fa-2b45-4988-9d05-b83a1fb3ba2e/volumes"
Oct 30 22:42:53.268034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:53.267997    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:54.370947 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.370916    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"]
Oct 30 22:42:54.371385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371368    4410 cpu_manager.go:414] "containerMap: removing container" podUID="7df2245e-8128-4b62-ad43-f6b802c36442" containerName="revoke-keys"
Oct 30 22:42:54.371385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371385    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df2245e-8128-4b62-ad43-f6b802c36442" containerName="revoke-keys"
Oct 30 22:42:54.371509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371409    4410 cpu_manager.go:414] "containerMap: removing container" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy"
Oct 30 22:42:54.371509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371415    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy"
Oct 30 22:42:54.371580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371538    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df2245e-8128-4b62-ad43-f6b802c36442" containerName="revoke-keys"
Oct 30 22:42:54.371580 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.371560    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb225fa-2b45-4988-9d05-b83a1fb3ba2e" containerName="istio-proxy"
Oct 30 22:42:54.376122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.376105    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.429703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.429682    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:54.479226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.479195    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:54.479447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.479390    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:54.479589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.479517    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:42:54.482313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.482289    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"]
Oct 30 22:42:54.507209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.507186    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8mv\" (UniqueName: \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.507332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.507310    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.507412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.507398    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.608319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.608289    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8mv\" (UniqueName: \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.608482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.608401    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.608556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.608494    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.609131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.609108    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.610765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.610747    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.666696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.666672    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8mv\" (UniqueName: \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\") pod \"maas-api-revoke-sub-541524d6-66mmw\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:54.689963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:54.689937    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:55.060050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:55.060025    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"]
Oct 30 22:42:55.061625 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:42:55.061585    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78022a8e_94ca_42d7_bbca_2264b8d6ec31.slice/crio-257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf WatchSource:0}: Error finding container 257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf: Status 404 returned error can't find the container with id 257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf
Oct 30 22:42:55.313168 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:55.313099    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw" event={"ID":"78022a8e-94ca-42d7-bbca-2264b8d6ec31","Type":"ContainerStarted","Data":"257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf"}
Oct 30 22:42:56.320643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:56.320563    4410 generic.go:356] "Generic (PLEG): container finished" podID="78022a8e-94ca-42d7-bbca-2264b8d6ec31" containerID="75f86767fdc5e259e10cba09187c0ec95111b4394bfc08055695acdf62b2c7d8" exitCode=0
Oct 30 22:42:56.321031 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:56.320663    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw" event={"ID":"78022a8e-94ca-42d7-bbca-2264b8d6ec31","Type":"ContainerDied","Data":"75f86767fdc5e259e10cba09187c0ec95111b4394bfc08055695acdf62b2c7d8"}
Oct 30 22:42:57.472391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.472360    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:57.638295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.638226    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\" (UniqueName: \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\") pod \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") "
Oct 30 22:42:57.638295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.638282    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\") pod \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") "
Oct 30 22:42:57.638479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.638399    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\") pod \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\" (UID: \"78022a8e-94ca-42d7-bbca-2264b8d6ec31\") "
Oct 30 22:42:57.638625 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.638605    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca" pod "78022a8e-94ca-42d7-bbca-2264b8d6ec31" (UID: "78022a8e-94ca-42d7-bbca-2264b8d6ec31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:57.638806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.638786    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:57.640258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.640234    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv" pod "78022a8e-94ca-42d7-bbca-2264b8d6ec31" (UID: "78022a8e-94ca-42d7-bbca-2264b8d6ec31"). InnerVolumeSpecName "kube-api-access-6m8mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:57.640415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.640394    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca" pod "78022a8e-94ca-42d7-bbca-2264b8d6ec31" (UID: "78022a8e-94ca-42d7-bbca-2264b8d6ec31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:57.739992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.739952    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78022a8e-94ca-42d7-bbca-2264b8d6ec31-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:57.739992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:57.739987    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8mv\" (UniqueName: \"kubernetes.io/projected/78022a8e-94ca-42d7-bbca-2264b8d6ec31-kube-api-access-6m8mv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:58.156075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:58.156044    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:58.333087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:58.333054    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"
Oct 30 22:42:58.333087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:58.333086    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw" event={"ID":"78022a8e-94ca-42d7-bbca-2264b8d6ec31","Type":"ContainerDied","Data":"257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf"}
Oct 30 22:42:58.333302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:42:58.333135    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257734aa182ceeff18a4f9362025f13bc0d69e6dd5d3b9a430cd469cc960b9cf"
Oct 30 22:43:03.042191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:03.042155    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:07.857308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:07.857257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:12.119977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:12.119881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:12.734722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:12.734689    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:17.610560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:17.610524    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:18.823241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:18.823204    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:20.265361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.265324    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:20.527894 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:43:20.527806    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:43:20.741129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.741094    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:43:20.741601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.741586    4410 cpu_manager.go:414] "containerMap: removing container" podUID="78022a8e-94ca-42d7-bbca-2264b8d6ec31" containerName="revoke-keys"
Oct 30 22:43:20.741647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.741602    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78022a8e-94ca-42d7-bbca-2264b8d6ec31" containerName="revoke-keys"
Oct 30 22:43:20.741712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.741701    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="78022a8e-94ca-42d7-bbca-2264b8d6ec31" containerName="revoke-keys"
Oct 30 22:43:20.745105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.745090    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.806888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.806178    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-15e0f6-openshift-default-dockercfg-4kdns\""
Oct 30 22:43:20.861248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.861206    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:20.861698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.861667    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:43:20.880869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.880839    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.880992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.880954    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881010    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4hv\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881034    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881126    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881162    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881192    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881207    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.881363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.881226    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982332    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982394    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982424    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982452    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982517    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982581    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982637    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4hv\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982673    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.982836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982755    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.983055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982863    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.983055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.982862    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.983179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.983137    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.983323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.983286    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.983526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.983506    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.985503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.985476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:20.985503 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:20.985492    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:21.039291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.039258    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4hv\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:21.039597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.039574    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\") pod \"e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") " pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:21.058920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.058859    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:21.142762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.142724    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:21.504167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.504143    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:43:21.505024 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:43:21.504993    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9b0fc7_7788_4ad5_b000_8a46c2964a2a.slice/crio-1e978992d25ba6549384cb7573449ac2ca0dceb809703e7d51069e5dee8c69c7 WatchSource:0}: Error finding container 1e978992d25ba6549384cb7573449ac2ca0dceb809703e7d51069e5dee8c69c7: Status 404 returned error can't find the container with id 1e978992d25ba6549384cb7573449ac2ca0dceb809703e7d51069e5dee8c69c7
Oct 30 22:43:21.507438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.507406    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:43:21.507527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.507467    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:43:21.507527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:21.507497    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:43:22.425391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:22.425361    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:22.471209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:22.471172    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerStarted","Data":"b8d3d76d53b291060ef07678a3a514ff6282732826171e8dc08fefee735ad894"}
Oct 30 22:43:22.471414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:22.471229    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerStarted","Data":"1e978992d25ba6549384cb7573449ac2ca0dceb809703e7d51069e5dee8c69c7"}
Oct 30 22:43:22.638753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:22.638693    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podStartSLOduration=2.638675688 podStartE2EDuration="2.638675688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:22.635266713 +0000 UTC m=+3655.724004516" watchObservedRunningTime="2026-10-30 22:43:22.638675688 +0000 UTC m=+3655.727413492"
Oct 30 22:43:23.059154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:23.059122    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:23.064725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:23.064699    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:23.477465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:23.477434    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:23.478478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:23.478456    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:43:25.425661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:25.425624    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:25.674047 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:25.674013    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.787908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:27.787877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.994970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:27.994912    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:28.168442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:28.168410    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:30.844256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:30.844220    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.044906 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.044877    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.230886 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.230854    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:43:31.235954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.235934    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.293107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.293081    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:43:31.294166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.294142    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.294299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.294269    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42nz\" (UniqueName: \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.339060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.338961    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-15e0f6-dockercfg-7vfgc\""
Oct 30 22:43:31.339060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.338959    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-15e0f6\""
Oct 30 22:43:31.395894 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.395856    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.396089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.395921    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42nz\" (UniqueName: \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.396584 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.396561    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.452113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.452086    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42nz\" (UniqueName: \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\") pod \"payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.549691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.549610    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:31.920803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:31.920763    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:43:31.922497 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:43:31.922468    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bd35f9_2cd4_4bc8_83da_492a94c83699.slice/crio-ff49d7edbb0caacb7245d37b73a2d08034e5244875a791b4ab40c3d9497f94fb WatchSource:0}: Error finding container ff49d7edbb0caacb7245d37b73a2d08034e5244875a791b4ab40c3d9497f94fb: Status 404 returned error can't find the container with id ff49d7edbb0caacb7245d37b73a2d08034e5244875a791b4ab40c3d9497f94fb
Oct 30 22:43:32.528614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:32.528576    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" event={"ID":"52bd35f9-2cd4-4bc8-83da-492a94c83699","Type":"ContainerStarted","Data":"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"}
Oct 30 22:43:32.528823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:32.528632    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" event={"ID":"52bd35f9-2cd4-4bc8-83da-492a94c83699","Type":"ContainerStarted","Data":"ff49d7edbb0caacb7245d37b73a2d08034e5244875a791b4ab40c3d9497f94fb"}
Oct 30 22:43:32.528823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:32.528673    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:32.645151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:32.645095    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" podStartSLOduration=1.6450814409999999 podStartE2EDuration="1.645081441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:32.641906788 +0000 UTC m=+3665.730644591" watchObservedRunningTime="2026-10-30 22:43:32.645081441 +0000 UTC m=+3665.733819261"
Oct 30 22:43:32.757570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:32.757531    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.541510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:34.541471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:37.673444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:37.673411    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:42.348724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:42.348681    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:42.510474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:42.510435    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:42.666494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:42.666459    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:43.537218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:43.537189    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:43:47.467868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:47.467828    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.522020 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:43:50.521974    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:43:52.388282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:52.388245    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:57.221242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:43:57.221208    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:02.028429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:02.028371    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:06.876652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:06.876616    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:11.659753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:11.659716    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.296755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:16.296713    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.603367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:16.603285    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:19.293837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.293797    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:19.803524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.803489    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:44:19.808153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.808128    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.860226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.860198    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-55edb1-openshift-default-dockercfg-p6cqp\""
Oct 30 22:44:19.907439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907396    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907482    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907576    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907703    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907764    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907868    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.907972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.907953    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.908029 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.908019    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhq6\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.908088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.908058    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:19.911613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.911585    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:19.913395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:19.913369    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:44:20.008766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.008732    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.008969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.008802    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.008969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.008850    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.008969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.008930    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.008969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.008966    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009014    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009054    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009088    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhq6\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009127    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009305    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009581    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.009827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.009805    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.010191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.010169    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.012069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.012047    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.012312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.012280    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.071027 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.070957    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.075355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.075326    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhq6\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\") pod \"e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") " pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.122183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.122153    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:20.339084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.338934    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:20.523202 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:20.523166    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:44:20.549686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.549458    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:44:20.552791 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:44:20.552742    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9638907f_d147_4133_8c65_45cf5c88feca.slice/crio-6776019031ee55a796a85275be4ce485bb507a02d6735ac7c5c0c4acc4bfc38b WatchSource:0}: Error finding container 6776019031ee55a796a85275be4ce485bb507a02d6735ac7c5c0c4acc4bfc38b: Status 404 returned error can't find the container with id 6776019031ee55a796a85275be4ce485bb507a02d6735ac7c5c0c4acc4bfc38b
Oct 30 22:44:20.555623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.555591    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:44:20.555719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.555658    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:44:20.555719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.555688    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:44:20.798969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.798931    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerStarted","Data":"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"}
Oct 30 22:44:20.799148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.798985    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerStarted","Data":"6776019031ee55a796a85275be4ce485bb507a02d6735ac7c5c0c4acc4bfc38b"}
Oct 30 22:44:20.923225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:20.923172    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podStartSLOduration=1.923157641 podStartE2EDuration="1.923157641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:20.918924875 +0000 UTC m=+3714.007662678" watchObservedRunningTime="2026-10-30 22:44:20.923157641 +0000 UTC m=+3714.011895496"
Oct 30 22:44:21.122783 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:21.122749    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:21.129143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:21.129117    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:21.504239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:21.504206    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:21.805252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:21.805177    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:21.806038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:21.806019    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:44:23.184357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:23.184314    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:23.746465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:23.746429    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:23.925450 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:23.925414    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:24.099169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:24.099093    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:26.786915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:26.786873    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.758639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:28.758588    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.950820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:28.950752    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:29.182911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.182879    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:44:29.186566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.186548    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.216021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.215997    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5ng\" (UniqueName: \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.216149 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.216132    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.293923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.293889    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-55edb1\""
Oct 30 22:44:29.295093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.293889    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-55edb1-dockercfg-w4rl4\""
Oct 30 22:44:29.295256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.295032    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:44:29.317547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.317514    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5ng\" (UniqueName: \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.317737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.317660    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.318370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.318349    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.375245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.375204    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5ng\" (UniqueName: \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\") pod \"payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.499751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.499675    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:29.873299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:29.873272    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:44:29.875115 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:44:29.875079    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d9bd92_e53b_493f_9293_408d0b9dd4f4.slice/crio-3aafe30f5b8e8637a898611d7a98533b572570ed43eb11f1a25947232ae9a1a9 WatchSource:0}: Error finding container 3aafe30f5b8e8637a898611d7a98533b572570ed43eb11f1a25947232ae9a1a9: Status 404 returned error can't find the container with id 3aafe30f5b8e8637a898611d7a98533b572570ed43eb11f1a25947232ae9a1a9
Oct 30 22:44:30.862790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:30.862747    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" event={"ID":"36d9bd92-e53b-493f-9293-408d0b9dd4f4","Type":"ContainerStarted","Data":"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"}
Oct 30 22:44:30.862979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:30.862832    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:30.862979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:30.862855    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" event={"ID":"36d9bd92-e53b-493f-9293-408d0b9dd4f4","Type":"ContainerStarted","Data":"3aafe30f5b8e8637a898611d7a98533b572570ed43eb11f1a25947232ae9a1a9"}
Oct 30 22:44:30.980849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:30.980770    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" podStartSLOduration=1.980751047 podStartE2EDuration="1.980751047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:30.97886997 +0000 UTC m=+3724.067607797" watchObservedRunningTime="2026-10-30 22:44:30.980751047 +0000 UTC m=+3724.069488851"
Oct 30 22:44:31.905923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:31.905885    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:34.896036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:34.896003    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:36.554172 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:36.554120    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:40.685413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.685333    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-6e7f54"]
Oct 30 22:44:40.689229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.689211    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:40.743993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.743961    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbdpx\""
Oct 30 22:44:40.746719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.746684    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6rr\" (UniqueName: \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\") pod \"test-curl-7373-6e7f54\" (UID: \"6817bdf9-9036-4731-a637-03e808d9190a\") " pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:40.747300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.747279    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-6e7f54"]
Oct 30 22:44:40.847867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.847835    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6rr\" (UniqueName: \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\") pod \"test-curl-7373-6e7f54\" (UID: \"6817bdf9-9036-4731-a637-03e808d9190a\") " pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:40.905988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:40.905963    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6rr\" (UniqueName: \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\") pod \"test-curl-7373-6e7f54\" (UID: \"6817bdf9-9036-4731-a637-03e808d9190a\") " pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:41.000965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.000894    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:41.374484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.374456    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-6e7f54"]
Oct 30 22:44:41.376382 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:44:41.376354    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6817bdf9_9036_4731_a637_03e808d9190a.slice/crio-25beb95f88b9646eec367cc1212d0fa747def84cb5b2f11fb0942aee2504aea3 WatchSource:0}: Error finding container 25beb95f88b9646eec367cc1212d0fa747def84cb5b2f11fb0942aee2504aea3: Status 404 returned error can't find the container with id 25beb95f88b9646eec367cc1212d0fa747def84cb5b2f11fb0942aee2504aea3
Oct 30 22:44:41.455682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.455654    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:41.870515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.870489    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:44:41.927633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.927603    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-6e7f54" event={"ID":"6817bdf9-9036-4731-a637-03e808d9190a","Type":"ContainerStarted","Data":"2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d"}
Oct 30 22:44:41.927803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:41.927640    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-6e7f54" event={"ID":"6817bdf9-9036-4731-a637-03e808d9190a","Type":"ContainerStarted","Data":"25beb95f88b9646eec367cc1212d0fa747def84cb5b2f11fb0942aee2504aea3"}
Oct 30 22:44:42.097757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:42.097699    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7373-6e7f54" podStartSLOduration=2.097685491 podStartE2EDuration="2.097685491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:42.097337057 +0000 UTC m=+3735.186074870" watchObservedRunningTime="2026-10-30 22:44:42.097685491 +0000 UTC m=+3735.186423294"
Oct 30 22:44:43.516471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.516433    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-6e7f54"]
Oct 30 22:44:43.523475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.523438    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-6e7f54"]
Oct 30 22:44:43.811642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.811567    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"]
Oct 30 22:44:43.812273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.812256    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6817bdf9-9036-4731-a637-03e808d9190a" containerName="test-curl-7373-6e7f54"
Oct 30 22:44:43.812331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.812276    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6817bdf9-9036-4731-a637-03e808d9190a" containerName="test-curl-7373-6e7f54"
Oct 30 22:44:43.812444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.812432    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6817bdf9-9036-4731-a637-03e808d9190a" containerName="test-curl-7373-6e7f54"
Oct 30 22:44:43.815958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.815937    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.870025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.869994    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:43.883048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.883018    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.883276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.883250    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.883350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.883298    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tg4\" (UniqueName: \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.919067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.919041    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:43.919187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.919127    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:44:43.919187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.919148    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:43.921979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.921959    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"]
Oct 30 22:44:43.938434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.938406    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7373-6e7f54" podUID="6817bdf9-9036-4731-a637-03e808d9190a" containerName="test-curl-7373-6e7f54" containerID="cri-o://2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d" gracePeriod=2
Oct 30 22:44:43.983985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.983960    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.984088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.983990    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tg4\" (UniqueName: \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.984088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.984045    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.984682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.984660    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:43.986228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:43.986207    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:44.019324 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:44.019298    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7373-6e7f54\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="6817bdf9-9036-4731-a637-03e808d9190a" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:44.041080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:44.041050    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tg4\" (UniqueName: \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\") pod \"maas-api-revoke-sub-6a64224a-trhlq\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:44.129712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:44.129658    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:44.503265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:44.503234    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"]
Oct 30 22:44:44.504989 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:44:44.504963    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb809f7d8_4601_414c_b518_8a28e82515fb.slice/crio-387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113 WatchSource:0}: Error finding container 387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113: Status 404 returned error can't find the container with id 387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113
Oct 30 22:44:44.945304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:44.945268    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq" event={"ID":"b809f7d8-4601-414c-b518-8a28e82515fb","Type":"ContainerStarted","Data":"ed5f4d849273212c55d9a4511543d46de9ab28141d553c0a4996a752b8fdf1c0"}
Oct 30 22:44:44.945759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:44.945327    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq" event={"ID":"b809f7d8-4601-414c-b518-8a28e82515fb","Type":"ContainerStarted","Data":"387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113"}
Oct 30 22:44:45.114937 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:45.114859    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7373-6e7f54\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="6817bdf9-9036-4731-a637-03e808d9190a" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:45.953938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:45.953910    4410 generic.go:356] "Generic (PLEG): container finished" podID="b809f7d8-4601-414c-b518-8a28e82515fb" containerID="ed5f4d849273212c55d9a4511543d46de9ab28141d553c0a4996a752b8fdf1c0" exitCode=0
Oct 30 22:44:45.954334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:45.953992    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq" event={"ID":"b809f7d8-4601-414c-b518-8a28e82515fb","Type":"ContainerDied","Data":"ed5f4d849273212c55d9a4511543d46de9ab28141d553c0a4996a752b8fdf1c0"}
Oct 30 22:44:46.170982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.170949    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:46.181872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.181834    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:46.309976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.309901    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\" (UniqueName: \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\") pod \"6817bdf9-9036-4731-a637-03e808d9190a\" (UID: \"6817bdf9-9036-4731-a637-03e808d9190a\") "
Oct 30 22:44:46.311917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.311893    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr" pod "6817bdf9-9036-4731-a637-03e808d9190a" (UID: "6817bdf9-9036-4731-a637-03e808d9190a"). InnerVolumeSpecName "kube-api-access-hf6rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:46.372908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.372875    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:46.411156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.411124    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6rr\" (UniqueName: \"kubernetes.io/projected/6817bdf9-9036-4731-a637-03e808d9190a-kube-api-access-hf6rr\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:46.531923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.531889    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:46.962717 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.962680    4410 generic.go:356] "Generic (PLEG): container finished" podID="6817bdf9-9036-4731-a637-03e808d9190a" containerID="2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d" exitCode=137
Oct 30 22:44:46.963188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.962736    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:46.963188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.962798    4410 scope.go:122] "RemoveContainer" containerID="2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d"
Oct 30 22:44:46.971984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.971962    4410 scope.go:122] "RemoveContainer" containerID="2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d"
Oct 30 22:44:46.972303 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:46.972282    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d\": container with ID starting with 2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d not found: ID does not exist" containerID="2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d"
Oct 30 22:44:46.972366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:46.972312    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d"} err="failed to get container status \"2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d\": rpc error: code = NotFound desc = could not find container \"2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d\": container with ID starting with 2383e8f41c76be50df4e7b96aee456e72231c9b6508fecb28cceccc85db0572d not found: ID does not exist"
Oct 30 22:44:47.020726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.020691    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6817bdf9-9036-4731-a637-03e808d9190a" path="/var/lib/kubelet/pods/6817bdf9-9036-4731-a637-03e808d9190a/volumes"
Oct 30 22:44:47.023101 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:47.023067    4410 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7373-6e7f54\" is forbidden: User \"system:node:ip-10-0-128-19.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-19.us-west-1.compute.internal' and this object" podUID="6817bdf9-9036-4731-a637-03e808d9190a" pod="opendatahub/test-curl-7373-6e7f54"
Oct 30 22:44:47.114004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.113979    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:47.218897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.218810    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\") pod \"b809f7d8-4601-414c-b518-8a28e82515fb\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") "
Oct 30 22:44:47.218897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.218858    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\") pod \"b809f7d8-4601-414c-b518-8a28e82515fb\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") "
Oct 30 22:44:47.219161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.218927    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\" (UniqueName: \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\") pod \"b809f7d8-4601-414c-b518-8a28e82515fb\" (UID: \"b809f7d8-4601-414c-b518-8a28e82515fb\") "
Oct 30 22:44:47.219250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.219223    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca" pod "b809f7d8-4601-414c-b518-8a28e82515fb" (UID: "b809f7d8-4601-414c-b518-8a28e82515fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:47.219523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.219494    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:47.220978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.220953    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca" pod "b809f7d8-4601-414c-b518-8a28e82515fb" (UID: "b809f7d8-4601-414c-b518-8a28e82515fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:47.221080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.220979    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4" pod "b809f7d8-4601-414c-b518-8a28e82515fb" (UID: "b809f7d8-4601-414c-b518-8a28e82515fb"). InnerVolumeSpecName "kube-api-access-q7tg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:47.320462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.320432    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b809f7d8-4601-414c-b518-8a28e82515fb-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:47.320462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.320459    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tg4\" (UniqueName: \"kubernetes.io/projected/b809f7d8-4601-414c-b518-8a28e82515fb-kube-api-access-q7tg4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:47.971609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.971577    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq" event={"ID":"b809f7d8-4601-414c-b518-8a28e82515fb","Type":"ContainerDied","Data":"387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113"}
Oct 30 22:44:47.972017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.971614    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"
Oct 30 22:44:47.972017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:47.971626    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387a5340835f99cff13d990a5e0f76638ca4997b9c653bcc7c1bae2dc0264113"
Oct 30 22:44:50.522701 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:44:50.522665    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:44:51.261673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:51.261641    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:54.568364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.568323    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"]
Oct 30 22:44:54.568820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.568807    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b809f7d8-4601-414c-b518-8a28e82515fb" containerName="revoke-keys"
Oct 30 22:44:54.568884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.568821    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b809f7d8-4601-414c-b518-8a28e82515fb" containerName="revoke-keys"
Oct 30 22:44:54.568936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.568919    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b809f7d8-4601-414c-b518-8a28e82515fb" containerName="revoke-keys"
Oct 30 22:44:54.572413 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.572395    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.629196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.629166    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"]
Oct 30 22:44:54.674672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.674639    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:54.674672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.674664    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:54.674895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.674698    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:54.674895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.674736    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:44:54.701600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.701577    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.701723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.701603    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjzt\" (UniqueName: \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.701723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.701624    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.802619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.802592    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.802619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.802620    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjzt\" (UniqueName: \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.802815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.802723    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.803365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.803340    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.805050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.805027    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.868291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.868230    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjzt\" (UniqueName: \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\") pod \"maas-api-revoke-sub-0316043a-pmggg\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:54.886010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:54.885990    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:55.267608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:55.267579    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"]
Oct 30 22:44:55.269236 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:44:55.269201    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3a96cc_db66_4c2f_bd91_d3414a2ca44f.slice/crio-712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b WatchSource:0}: Error finding container 712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b: Status 404 returned error can't find the container with id 712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b
Oct 30 22:44:56.019416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:56.019380    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg" event={"ID":"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f","Type":"ContainerStarted","Data":"887aabc00bfc95ff42a4196b3a9cae167be8d98702d1ab0c463bb302179ee53e"}
Oct 30 22:44:56.019847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:56.019434    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg" event={"ID":"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f","Type":"ContainerStarted","Data":"712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b"}
Oct 30 22:44:56.135673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:56.135611    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg" podStartSLOduration=2.135593444 podStartE2EDuration="2.135593444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:56.131623371 +0000 UTC m=+3749.220361175" watchObservedRunningTime="2026-10-30 22:44:56.135593444 +0000 UTC m=+3749.224331248"
Oct 30 22:44:56.151551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:56.151528    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.920794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:56.920730    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:57.026213 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:57.026173    4410 generic.go:356] "Generic (PLEG): container finished" podID="fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" containerID="887aabc00bfc95ff42a4196b3a9cae167be8d98702d1ab0c463bb302179ee53e" exitCode=0
Oct 30 22:44:57.026626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:57.026211    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg" event={"ID":"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f","Type":"ContainerDied","Data":"887aabc00bfc95ff42a4196b3a9cae167be8d98702d1ab0c463bb302179ee53e"}
Oct 30 22:44:58.178137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.178112    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:44:58.344091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.344005    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\" (UniqueName: \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\") pod \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") "
Oct 30 22:44:58.344302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.344135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\") pod \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") "
Oct 30 22:44:58.344302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.344186    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\") pod \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\" (UID: \"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f\") "
Oct 30 22:44:58.344499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.344475    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca" pod "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" (UID: "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:58.346166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.346138    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca" pod "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" (UID: "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:58.346262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.346147    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt" pod "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" (UID: "fa3a96cc-db66-4c2f-bd91-d3414a2ca44f"). InnerVolumeSpecName "kube-api-access-ptjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:58.446055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.446024    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:58.446055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.446052    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjzt\" (UniqueName: \"kubernetes.io/projected/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-kube-api-access-ptjzt\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:58.446055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:58.446063    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:59.038916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:59.038886    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg" event={"ID":"fa3a96cc-db66-4c2f-bd91-d3414a2ca44f","Type":"ContainerDied","Data":"712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b"}
Oct 30 22:44:59.039055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:59.038935    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="712b63933cc6fe4d0865f0f4cf887475591b7cc80b36f49141a9f42e110a976b"
Oct 30 22:44:59.039055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:44:59.038894    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"
Oct 30 22:45:00.160649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.160621    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:00.161690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.161650    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" containerName="revoke-keys"
Oct 30 22:45:00.161851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.161692    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" containerName="revoke-keys"
Oct 30 22:45:00.162075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.162059    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" containerName="revoke-keys"
Oct 30 22:45:00.172300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.172276    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:00.172485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.172455    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.177367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.177342    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:00.177505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.177452    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.180953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.180935    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.209818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.209792    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:00.213348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.213332    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.219848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.219824    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:00.224106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.224086    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:00.224227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.224207    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.228542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.228521    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:00.228653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.228628    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.232869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.232852    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:00.232998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.232980    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.236860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.236840    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:00.236997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.236909    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.240482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.240461    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:00.240585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.240555    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.244274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.244256    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.260602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.259679    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:00.262243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262217    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.262355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262257    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.262355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262282    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r64tm\" (UniqueName: \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.262479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262386    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.262479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262431    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25jv\" (UniqueName: \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.262615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262498    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.262615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262532    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.262615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262568    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.262615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262610    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s9d\" (UniqueName: \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.262866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262712    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2crt\" (UniqueName: \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.262866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262765    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmgb\" (UniqueName: \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.262866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262843    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.263025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262884    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.263025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262913    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.263025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.262945    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8jz\" (UniqueName: \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.263192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.263014    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nksrr\" (UniqueName: \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.263192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.263091    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxxl\" (UniqueName: \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.263192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.263134    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.263192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.263178    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrjl\" (UniqueName: \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.263390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.263215    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75z7\" (UniqueName: \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.264953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.264933    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:45:00.265062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.264959    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.265062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.265030    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.265232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.265216    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:00.265316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.265305    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.268995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.268974    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:00.272135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.272110    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:00.278758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.276720    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:00.280683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.280652    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:00.313120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.313095    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:00.316883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.316852    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:00.329633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.327907    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:00.331833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.331809    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:00.339155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.339135    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:00.364809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364753    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.364927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364815    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64tm\" (UniqueName: \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.364927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364845    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.364927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364868    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25jv\" (UniqueName: \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.364927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364885    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.364927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364909    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.365211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.364956    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.365544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365521    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.365630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365523    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.365737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365714    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.365843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365730    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.365843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365714    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.365843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365799    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s9d\" (UniqueName: \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.365843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365842    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2crt\" (UniqueName: \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365869    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmgb\" (UniqueName: \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365891    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365923    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365961    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schdw\" (UniqueName: \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.365990    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.366041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366029    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8jz\" (UniqueName: \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.366298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366085    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nksrr\" (UniqueName: \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.366298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366175    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxxl\" (UniqueName: \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.366298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366217    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.366298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366258    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.366523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366382    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrjl\" (UniqueName: \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.366523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366429    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75z7\" (UniqueName: \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.366523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366474    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.366693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366671    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.366761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366671    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.366761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366715    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.366943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.366926    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.367102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.367086    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.380112 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.380091    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:00.425427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.425373    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25jv\" (UniqueName: \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\") pod \"maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.425557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.425454    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nksrr\" (UniqueName: \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\") pod \"maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.425687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.425662    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s9d\" (UniqueName: \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.426212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.426189    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64tm\" (UniqueName: \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\") pod \"maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.426311 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.426261    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmgb\" (UniqueName: \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.426444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.426358    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxxl\" (UniqueName: \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.426595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.426572    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2crt\" (UniqueName: \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\") pod \"maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.426757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.426740    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75z7\" (UniqueName: \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.427353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.427336    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrjl\" (UniqueName: \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.428268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.428250    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8jz\" (UniqueName: \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\") pod \"maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.467839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.467802    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schdw\" (UniqueName: \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.467963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.467882    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.468430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.468409    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.487517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.487496    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:00.495134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.495112    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:00.501026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.501009    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:00.525051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.525020    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schdw\" (UniqueName: \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\") pod \"maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.527508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.527479    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:00.541978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.541574    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:00.546743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.546710    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:00.563919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.557976    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:00.568184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.568002    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:00.587806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.587757    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:00.598431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.598403    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:00.608756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.608729    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:00.880121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.879975    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:00.883224 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:00.883198    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16dda5b_a4dd_45b9_941d_45d09d307557.slice/crio-37db58c4c18dcc965ffdf0d330efff2fb74b2b9690eca0c653e72852c23f7653 WatchSource:0}: Error finding container 37db58c4c18dcc965ffdf0d330efff2fb74b2b9690eca0c653e72852c23f7653: Status 404 returned error can't find the container with id 37db58c4c18dcc965ffdf0d330efff2fb74b2b9690eca0c653e72852c23f7653
Oct 30 22:45:00.932824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.932786    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:00.933137 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:00.933111    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f02fb68_1152_4976_a9df_e607572f5e3b.slice/crio-79fdf81cacfcf6fe89324945ea1beb9d057a353335b08392e26c158582202c74 WatchSource:0}: Error finding container 79fdf81cacfcf6fe89324945ea1beb9d057a353335b08392e26c158582202c74: Status 404 returned error can't find the container with id 79fdf81cacfcf6fe89324945ea1beb9d057a353335b08392e26c158582202c74
Oct 30 22:45:00.935685 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:00.935442    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085629ab_4bfd_461e_beac_0ad6a2cacbb6.slice/crio-961e156a6f7c48450277d1890d58cb514d596502a212938e403208555cefc165 WatchSource:0}: Error finding container 961e156a6f7c48450277d1890d58cb514d596502a212938e403208555cefc165: Status 404 returned error can't find the container with id 961e156a6f7c48450277d1890d58cb514d596502a212938e403208555cefc165
Oct 30 22:45:00.944419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:00.937531    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:01.009412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.009139    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:01.011616 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.011588    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77d9452_8280_41b9_b8e0_c702d516e15d.slice/crio-7f93ccf17694b41c48f15a5575bcb21dc6c0710b1203deac16f0fd8f5c29bdfc WatchSource:0}: Error finding container 7f93ccf17694b41c48f15a5575bcb21dc6c0710b1203deac16f0fd8f5c29bdfc: Status 404 returned error can't find the container with id 7f93ccf17694b41c48f15a5575bcb21dc6c0710b1203deac16f0fd8f5c29bdfc
Oct 30 22:45:01.012302 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.012277    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode915b9b8_f770_4dd8_bb14_1f5ca60aba2e.slice/crio-4f4349d668c12eb7c843c1d97c5e3154d2988d8ac116b3fd19998dce7be04445 WatchSource:0}: Error finding container 4f4349d668c12eb7c843c1d97c5e3154d2988d8ac116b3fd19998dce7be04445: Status 404 returned error can't find the container with id 4f4349d668c12eb7c843c1d97c5e3154d2988d8ac116b3fd19998dce7be04445
Oct 30 22:45:01.021393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.021351    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:01.042910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.042889    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:01.068200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.068056    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerStarted","Data":"79fdf81cacfcf6fe89324945ea1beb9d057a353335b08392e26c158582202c74"}
Oct 30 22:45:01.069601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.069577    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerStarted","Data":"37db58c4c18dcc965ffdf0d330efff2fb74b2b9690eca0c653e72852c23f7653"}
Oct 30 22:45:01.071449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.071420    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerStarted","Data":"961e156a6f7c48450277d1890d58cb514d596502a212938e403208555cefc165"}
Oct 30 22:45:01.073388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.073364    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerStarted","Data":"4f4349d668c12eb7c843c1d97c5e3154d2988d8ac116b3fd19998dce7be04445"}
Oct 30 22:45:01.074491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.074469    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerStarted","Data":"7f93ccf17694b41c48f15a5575bcb21dc6c0710b1203deac16f0fd8f5c29bdfc"}
Oct 30 22:45:01.089319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.089297    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:01.097337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.097319    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:01.101455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.101436    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:01.107591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.107569    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:01.149028 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.148992    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ce4291_9682_4b75_8079_a7502f4c5816.slice/crio-e19fd0f244a5dcf6b7301ae240fdfc3f836f06ce9cbb182239597697f79a2094 WatchSource:0}: Error finding container e19fd0f244a5dcf6b7301ae240fdfc3f836f06ce9cbb182239597697f79a2094: Status 404 returned error can't find the container with id e19fd0f244a5dcf6b7301ae240fdfc3f836f06ce9cbb182239597697f79a2094
Oct 30 22:45:01.150376 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.150352    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd760d22b_efd0_490f_b4f1_6d91c7d5da85.slice/crio-fd6e33b95e8c5b0d526b365c5e68b959040ede0263a6852e9621695952dcfc7e WatchSource:0}: Error finding container fd6e33b95e8c5b0d526b365c5e68b959040ede0263a6852e9621695952dcfc7e: Status 404 returned error can't find the container with id fd6e33b95e8c5b0d526b365c5e68b959040ede0263a6852e9621695952dcfc7e
Oct 30 22:45:01.248094 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.248023    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf67d3b_c9ed_4047_914d_cc1723471db3.slice/crio-ad7225f7450b009cdd10fb2423ba41286dc25b6197d1facf1d8cfdfbeb7cb30b WatchSource:0}: Error finding container ad7225f7450b009cdd10fb2423ba41286dc25b6197d1facf1d8cfdfbeb7cb30b: Status 404 returned error can't find the container with id ad7225f7450b009cdd10fb2423ba41286dc25b6197d1facf1d8cfdfbeb7cb30b
Oct 30 22:45:01.249911 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.249863    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6de6f8_5a01_483c_abd5_4a1af9488ee7.slice/crio-c35c3847acb3fb55a9e9b9498940ce8fa20b87420c4ddae7e86b745602a8d209 WatchSource:0}: Error finding container c35c3847acb3fb55a9e9b9498940ce8fa20b87420c4ddae7e86b745602a8d209: Status 404 returned error can't find the container with id c35c3847acb3fb55a9e9b9498940ce8fa20b87420c4ddae7e86b745602a8d209
Oct 30 22:45:01.287175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.287146    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:01.411114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:01.411076    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:01.449443 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:01.449391    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659fe955_2d44_4057_8a77_974edb3ec5b9.slice/crio-dcc4e4754b04cd971c0cb14805fa363c14a245bc45f87dc1e6efd63fb8be796a WatchSource:0}: Error finding container dcc4e4754b04cd971c0cb14805fa363c14a245bc45f87dc1e6efd63fb8be796a: Status 404 returned error can't find the container with id dcc4e4754b04cd971c0cb14805fa363c14a245bc45f87dc1e6efd63fb8be796a
Oct 30 22:45:02.082033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.081986    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerStarted","Data":"289cf42f9fe12cfd90a3345e21b93683c8c63d059d476f020d7d4dc7fa986020"}
Oct 30 22:45:02.083522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.083500    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerStarted","Data":"b7473cb65b366bd1ddad2be0147c94749d1887e9f651ede685f16a47c1e16a3c"}
Oct 30 22:45:02.083638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.083544    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerStarted","Data":"e19fd0f244a5dcf6b7301ae240fdfc3f836f06ce9cbb182239597697f79a2094"}
Oct 30 22:45:02.085151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.085124    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerStarted","Data":"3ae14c8a5e03cf7d7798c37561036d9c0730f1557d685769d809c518a7710f03"}
Oct 30 22:45:02.085256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.085169    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerStarted","Data":"fd6e33b95e8c5b0d526b365c5e68b959040ede0263a6852e9621695952dcfc7e"}
Oct 30 22:45:02.086558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.086535    4410 generic.go:356] "Generic (PLEG): container finished" podID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerID="a7f855d2d6c366d19e2a171078d8c3ded3e659324959a5008992c96b07655112" exitCode=60
Oct 30 22:45:02.086650 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.086601    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerDied","Data":"a7f855d2d6c366d19e2a171078d8c3ded3e659324959a5008992c96b07655112"}
Oct 30 22:45:02.086879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.086857    4410 scope.go:122] "RemoveContainer" containerID="a7f855d2d6c366d19e2a171078d8c3ded3e659324959a5008992c96b07655112"
Oct 30 22:45:02.088088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.088065    4410 generic.go:356] "Generic (PLEG): container finished" podID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerID="6e696bd49770a5e3f8b7a94cc9ceb6eb4735d7c193bd1a217018092f91a3ce5e" exitCode=60
Oct 30 22:45:02.088200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.088153    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerDied","Data":"6e696bd49770a5e3f8b7a94cc9ceb6eb4735d7c193bd1a217018092f91a3ce5e"}
Oct 30 22:45:02.088321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.088303    4410 scope.go:122] "RemoveContainer" containerID="6e696bd49770a5e3f8b7a94cc9ceb6eb4735d7c193bd1a217018092f91a3ce5e"
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.089935    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerStarted","Data":"4f49bb69abb79fb05b957de6e3a9a2e24e88d0e9b10fe715e0b57ac76744c5d8"}
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.089984    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerStarted","Data":"ad7225f7450b009cdd10fb2423ba41286dc25b6197d1facf1d8cfdfbeb7cb30b"}
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.091484    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerStarted","Data":"770d3ab5591d03fdff052e49b973744791fd16b041cd71171cc2feec655e1547"}
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.091520    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerStarted","Data":"dcc4e4754b04cd971c0cb14805fa363c14a245bc45f87dc1e6efd63fb8be796a"}
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.092930    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerStarted","Data":"a6a740fbf062fee4b82b0bb0906abeb21eebdfee76e47a9703768a14100fef42"}
Oct 30 22:45:02.093058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.092960    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerStarted","Data":"98d2089726a78ddff16a279222f857d045459c64f0b98034fe3bdd237973b544"}
Oct 30 22:45:02.094489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.094464    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerStarted","Data":"4d013eca726b1ab15cb0944a12dd4fe27bc303fde0192e418606d2feeffb91ae"}
Oct 30 22:45:02.094599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.094508    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerStarted","Data":"c35c3847acb3fb55a9e9b9498940ce8fa20b87420c4ddae7e86b745602a8d209"}
Oct 30 22:45:02.095955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.095926    4410 generic.go:356] "Generic (PLEG): container finished" podID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerID="186a9ebf8787d90ddffd4aec4c9070ad4d58951f86604f82f67beb3feaa60581" exitCode=60
Oct 30 22:45:02.096045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.095999    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerDied","Data":"186a9ebf8787d90ddffd4aec4c9070ad4d58951f86604f82f67beb3feaa60581"}
Oct 30 22:45:02.096147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.096131    4410 scope.go:122] "RemoveContainer" containerID="186a9ebf8787d90ddffd4aec4c9070ad4d58951f86604f82f67beb3feaa60581"
Oct 30 22:45:02.097680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.097652    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerStarted","Data":"554bcd24ba179379d057f0c81d5fbfc8568b8adeb3aea295327f7cf134158a7d"}
Oct 30 22:45:02.211088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.211031    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" podStartSLOduration=2.211012748 podStartE2EDuration="2.211012748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.210366551 +0000 UTC m=+3755.299104357" watchObservedRunningTime="2026-10-30 22:45:02.211012748 +0000 UTC m=+3755.299750554"
Oct 30 22:45:02.450093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.449433    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" podStartSLOduration=2.449399325 podStartE2EDuration="2.449399325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.447767666 +0000 UTC m=+3755.536505470" watchObservedRunningTime="2026-10-30 22:45:02.449399325 +0000 UTC m=+3755.538137129"
Oct 30 22:45:02.562860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.562815    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" podStartSLOduration=2.562799485 podStartE2EDuration="2.562799485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.561253753 +0000 UTC m=+3755.649991569" watchObservedRunningTime="2026-10-30 22:45:02.562799485 +0000 UTC m=+3755.651537287"
Oct 30 22:45:02.676598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:02.676543    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" podStartSLOduration=2.676523869 podStartE2EDuration="2.676523869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.672424131 +0000 UTC m=+3755.761161936" watchObservedRunningTime="2026-10-30 22:45:02.676523869 +0000 UTC m=+3755.765261673"
Oct 30 22:45:03.010966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.010910    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" podStartSLOduration=3.010893153 podStartE2EDuration="3.010893153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.010335213 +0000 UTC m=+3756.099073016" watchObservedRunningTime="2026-10-30 22:45:03.010893153 +0000 UTC m=+3756.099630959"
Oct 30 22:45:03.105985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.105952    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerStarted","Data":"9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"}
Oct 30 22:45:03.107829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.107800    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerStarted","Data":"e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"}
Oct 30 22:45:03.109330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.109304    4410 generic.go:356] "Generic (PLEG): container finished" podID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerID="4f49bb69abb79fb05b957de6e3a9a2e24e88d0e9b10fe715e0b57ac76744c5d8" exitCode=60
Oct 30 22:45:03.109414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.109384    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerDied","Data":"4f49bb69abb79fb05b957de6e3a9a2e24e88d0e9b10fe715e0b57ac76744c5d8"}
Oct 30 22:45:03.109577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.109561    4410 scope.go:122] "RemoveContainer" containerID="4f49bb69abb79fb05b957de6e3a9a2e24e88d0e9b10fe715e0b57ac76744c5d8"
Oct 30 22:45:03.110899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.110880    4410 generic.go:356] "Generic (PLEG): container finished" podID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerID="770d3ab5591d03fdff052e49b973744791fd16b041cd71171cc2feec655e1547" exitCode=60
Oct 30 22:45:03.111020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.110959    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerDied","Data":"770d3ab5591d03fdff052e49b973744791fd16b041cd71171cc2feec655e1547"}
Oct 30 22:45:03.111148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.111132    4410 scope.go:122] "RemoveContainer" containerID="770d3ab5591d03fdff052e49b973744791fd16b041cd71171cc2feec655e1547"
Oct 30 22:45:03.112883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.112710    4410 generic.go:356] "Generic (PLEG): container finished" podID="2b2940da-362c-48ec-a17c-36b85e65d321" containerID="a6a740fbf062fee4b82b0bb0906abeb21eebdfee76e47a9703768a14100fef42" exitCode=60
Oct 30 22:45:03.112883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.112795    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerDied","Data":"a6a740fbf062fee4b82b0bb0906abeb21eebdfee76e47a9703768a14100fef42"}
Oct 30 22:45:03.113026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.113004    4410 scope.go:122] "RemoveContainer" containerID="a6a740fbf062fee4b82b0bb0906abeb21eebdfee76e47a9703768a14100fef42"
Oct 30 22:45:03.114899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.114877    4410 generic.go:356] "Generic (PLEG): container finished" podID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerID="4d013eca726b1ab15cb0944a12dd4fe27bc303fde0192e418606d2feeffb91ae" exitCode=60
Oct 30 22:45:03.115007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.114991    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerDied","Data":"4d013eca726b1ab15cb0944a12dd4fe27bc303fde0192e418606d2feeffb91ae"}
Oct 30 22:45:03.115234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.115221    4410 scope.go:122] "RemoveContainer" containerID="4d013eca726b1ab15cb0944a12dd4fe27bc303fde0192e418606d2feeffb91ae"
Oct 30 22:45:03.117608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.117581    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerStarted","Data":"19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"}
Oct 30 22:45:03.119114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.119091    4410 generic.go:356] "Generic (PLEG): container finished" podID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerID="554bcd24ba179379d057f0c81d5fbfc8568b8adeb3aea295327f7cf134158a7d" exitCode=60
Oct 30 22:45:03.119216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.119162    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerDied","Data":"554bcd24ba179379d057f0c81d5fbfc8568b8adeb3aea295327f7cf134158a7d"}
Oct 30 22:45:03.119313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.119297    4410 scope.go:122] "RemoveContainer" containerID="554bcd24ba179379d057f0c81d5fbfc8568b8adeb3aea295327f7cf134158a7d"
Oct 30 22:45:03.121499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.121478    4410 generic.go:356] "Generic (PLEG): container finished" podID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerID="289cf42f9fe12cfd90a3345e21b93683c8c63d059d476f020d7d4dc7fa986020" exitCode=60
Oct 30 22:45:03.121590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.121558    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerDied","Data":"289cf42f9fe12cfd90a3345e21b93683c8c63d059d476f020d7d4dc7fa986020"}
Oct 30 22:45:03.121708 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.121695    4410 scope.go:122] "RemoveContainer" containerID="289cf42f9fe12cfd90a3345e21b93683c8c63d059d476f020d7d4dc7fa986020"
Oct 30 22:45:03.123183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.123162    4410 generic.go:356] "Generic (PLEG): container finished" podID="75ce4291-9682-4b75-8079-a7502f4c5816" containerID="b7473cb65b366bd1ddad2be0147c94749d1887e9f651ede685f16a47c1e16a3c" exitCode=60
Oct 30 22:45:03.123308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.123252    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerDied","Data":"b7473cb65b366bd1ddad2be0147c94749d1887e9f651ede685f16a47c1e16a3c"}
Oct 30 22:45:03.123534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.123515    4410 scope.go:122] "RemoveContainer" containerID="b7473cb65b366bd1ddad2be0147c94749d1887e9f651ede685f16a47c1e16a3c"
Oct 30 22:45:03.125815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.124506    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" podStartSLOduration=3.124491637 podStartE2EDuration="3.124491637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.123988942 +0000 UTC m=+3756.212726761" watchObservedRunningTime="2026-10-30 22:45:03.124491637 +0000 UTC m=+3756.213229441"
Oct 30 22:45:03.127177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.127149    4410 generic.go:356] "Generic (PLEG): container finished" podID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerID="3ae14c8a5e03cf7d7798c37561036d9c0730f1557d685769d809c518a7710f03" exitCode=60
Oct 30 22:45:03.127278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.127212    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerDied","Data":"3ae14c8a5e03cf7d7798c37561036d9c0730f1557d685769d809c518a7710f03"}
Oct 30 22:45:03.127446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.127428    4410 scope.go:122] "RemoveContainer" containerID="3ae14c8a5e03cf7d7798c37561036d9c0730f1557d685769d809c518a7710f03"
Oct 30 22:45:03.241086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.241033    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" podStartSLOduration=3.241018885 podStartE2EDuration="3.241018885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.238636073 +0000 UTC m=+3756.327373877" watchObservedRunningTime="2026-10-30 22:45:03.241018885 +0000 UTC m=+3756.329756688"
Oct 30 22:45:03.355392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.355334    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" podStartSLOduration=3.355317395 podStartE2EDuration="3.355317395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.354480443 +0000 UTC m=+3756.443218245" watchObservedRunningTime="2026-10-30 22:45:03.355317395 +0000 UTC m=+3756.444055200"
Oct 30 22:45:03.579102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.579059    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" podStartSLOduration=3.579044203 podStartE2EDuration="3.579044203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.577420115 +0000 UTC m=+3756.666157931" watchObservedRunningTime="2026-10-30 22:45:03.579044203 +0000 UTC m=+3756.667782038"
Oct 30 22:45:03.690664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:03.690618    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" podStartSLOduration=3.690603319 podStartE2EDuration="3.690603319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.689127494 +0000 UTC m=+3756.777865300" watchObservedRunningTime="2026-10-30 22:45:03.690603319 +0000 UTC m=+3756.779341122"
Oct 30 22:45:04.136514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.136415    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerStarted","Data":"de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"}
Oct 30 22:45:04.138693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.138663    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerStarted","Data":"ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"}
Oct 30 22:45:04.140566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.140540    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerStarted","Data":"f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"}
Oct 30 22:45:04.142378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.142345    4410 generic.go:356] "Generic (PLEG): container finished" podID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerID="19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694" exitCode=60
Oct 30 22:45:04.142459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.142389    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerDied","Data":"19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"}
Oct 30 22:45:04.142459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.142446    4410 scope.go:122] "RemoveContainer" containerID="186a9ebf8787d90ddffd4aec4c9070ad4d58951f86604f82f67beb3feaa60581"
Oct 30 22:45:04.142653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.142639    4410 scope.go:122] "RemoveContainer" containerID="19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"
Oct 30 22:45:04.142930 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:04.142907    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp_odh-ai-gateway-infra(085629ab-4bfd-461e-beac-0ad6a2cacbb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6"
Oct 30 22:45:04.144361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.144340    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerStarted","Data":"0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"}
Oct 30 22:45:04.148949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.148927    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerStarted","Data":"988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"}
Oct 30 22:45:04.151898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.151864    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerStarted","Data":"b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"}
Oct 30 22:45:04.155722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.155694    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerStarted","Data":"b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"}
Oct 30 22:45:04.157760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.157733    4410 generic.go:356] "Generic (PLEG): container finished" podID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerID="9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5" exitCode=60
Oct 30 22:45:04.157893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.157824    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerDied","Data":"9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"}
Oct 30 22:45:04.158074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.158056    4410 scope.go:122] "RemoveContainer" containerID="9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"
Oct 30 22:45:04.158303 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:04.158280    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg_odh-ai-gateway-infra(2f02fb68-1152-4976-a9df-e607572f5e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b"
Oct 30 22:45:04.160037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.160017    4410 generic.go:356] "Generic (PLEG): container finished" podID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerID="e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6" exitCode=60
Oct 30 22:45:04.160144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.160076    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerDied","Data":"e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"}
Oct 30 22:45:04.160329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.160264    4410 scope.go:122] "RemoveContainer" containerID="e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"
Oct 30 22:45:04.160545 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:04.160435    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x_odh-ai-gateway-infra(b16dda5b-a4dd-45b9-941d-45d09d307557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557"
Oct 30 22:45:04.162316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.162297    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerStarted","Data":"ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"}
Oct 30 22:45:04.321087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.321065    4410 scope.go:122] "RemoveContainer" containerID="a7f855d2d6c366d19e2a171078d8c3ded3e659324959a5008992c96b07655112"
Oct 30 22:45:04.333913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.333891    4410 scope.go:122] "RemoveContainer" containerID="6e696bd49770a5e3f8b7a94cc9ceb6eb4735d7c193bd1a217018092f91a3ce5e"
Oct 30 22:45:04.363030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:04.362985    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" podStartSLOduration=4.362971079 podStartE2EDuration="4.362971079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.361165033 +0000 UTC m=+3757.449902839" watchObservedRunningTime="2026-10-30 22:45:04.362971079 +0000 UTC m=+3757.451708934"
Oct 30 22:45:05.171397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.171366    4410 scope.go:122] "RemoveContainer" containerID="9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"
Oct 30 22:45:05.171821 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.171663    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg_odh-ai-gateway-infra(2f02fb68-1152-4976-a9df-e607572f5e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b"
Oct 30 22:45:05.173060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.173035    4410 scope.go:122] "RemoveContainer" containerID="e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"
Oct 30 22:45:05.173308 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.173285    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x_odh-ai-gateway-infra(b16dda5b-a4dd-45b9-941d-45d09d307557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557"
Oct 30 22:45:05.174646 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.174622    4410 generic.go:356] "Generic (PLEG): container finished" podID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerID="ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3" exitCode=60
Oct 30 22:45:05.174740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.174688    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerDied","Data":"ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"}
Oct 30 22:45:05.174740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.174734    4410 scope.go:122] "RemoveContainer" containerID="4f49bb69abb79fb05b957de6e3a9a2e24e88d0e9b10fe715e0b57ac76744c5d8"
Oct 30 22:45:05.174994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.174974    4410 scope.go:122] "RemoveContainer" containerID="ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"
Oct 30 22:45:05.175251 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.175223    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658_odh-ai-gateway-infra(cdf67d3b-c9ed-4047-914d-cc1723471db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3"
Oct 30 22:45:05.176621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.176598    4410 generic.go:356] "Generic (PLEG): container finished" podID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerID="de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a" exitCode=60
Oct 30 22:45:05.176716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.176681    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerDied","Data":"de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"}
Oct 30 22:45:05.176918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.176892    4410 scope.go:122] "RemoveContainer" containerID="de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"
Oct 30 22:45:05.177148 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.177130    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m_odh-ai-gateway-infra(659fe955-2d44-4057-8a77-974edb3ec5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9"
Oct 30 22:45:05.178419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.178397    4410 generic.go:356] "Generic (PLEG): container finished" podID="2b2940da-362c-48ec-a17c-36b85e65d321" containerID="ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f" exitCode=60
Oct 30 22:45:05.178496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.178474    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerDied","Data":"ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"}
Oct 30 22:45:05.178671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.178654    4410 scope.go:122] "RemoveContainer" containerID="ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"
Oct 30 22:45:05.178926 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.178900    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh_odh-ai-gateway-infra(2b2940da-362c-48ec-a17c-36b85e65d321)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" podUID="2b2940da-362c-48ec-a17c-36b85e65d321"
Oct 30 22:45:05.180480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.180460    4410 generic.go:356] "Generic (PLEG): container finished" podID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerID="f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93" exitCode=60
Oct 30 22:45:05.180636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.180542    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerDied","Data":"f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"}
Oct 30 22:45:05.180763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.180746    4410 scope.go:122] "RemoveContainer" containerID="f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"
Oct 30 22:45:05.181034 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.181011    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq_odh-ai-gateway-infra(ad6de6f8-5a01-483c-abd5-4a1af9488ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7"
Oct 30 22:45:05.182440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.182420    4410 scope.go:122] "RemoveContainer" containerID="19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"
Oct 30 22:45:05.182660 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.182635    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp_odh-ai-gateway-infra(085629ab-4bfd-461e-beac-0ad6a2cacbb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6"
Oct 30 22:45:05.183973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.183953    4410 generic.go:356] "Generic (PLEG): container finished" podID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerID="0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2" exitCode=60
Oct 30 22:45:05.184056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.184025    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerDied","Data":"0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"}
Oct 30 22:45:05.184194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.184181    4410 scope.go:122] "RemoveContainer" containerID="0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"
Oct 30 22:45:05.184401 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.184375    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75_odh-ai-gateway-infra(e915b9b8-f770-4dd8-bb14-1f5ca60aba2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e"
Oct 30 22:45:05.185924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.185900    4410 generic.go:356] "Generic (PLEG): container finished" podID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerID="988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7" exitCode=60
Oct 30 22:45:05.185995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.185978    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerDied","Data":"988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"}
Oct 30 22:45:05.186136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.186121    4410 scope.go:122] "RemoveContainer" containerID="988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"
Oct 30 22:45:05.186334 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.186303    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn_odh-ai-gateway-infra(b77d9452-8280-41b9-b8e0-c702d516e15d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d"
Oct 30 22:45:05.191208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.191189    4410 scope.go:122] "RemoveContainer" containerID="770d3ab5591d03fdff052e49b973744791fd16b041cd71171cc2feec655e1547"
Oct 30 22:45:05.191279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.191260    4410 generic.go:356] "Generic (PLEG): container finished" podID="75ce4291-9682-4b75-8079-a7502f4c5816" containerID="b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1" exitCode=60
Oct 30 22:45:05.191359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.191342    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerDied","Data":"b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"}
Oct 30 22:45:05.191552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.191539    4410 scope.go:122] "RemoveContainer" containerID="b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"
Oct 30 22:45:05.191729 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.191712    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb_odh-ai-gateway-infra(75ce4291-9682-4b75-8079-a7502f4c5816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" podUID="75ce4291-9682-4b75-8079-a7502f4c5816"
Oct 30 22:45:05.193400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.193379    4410 generic.go:356] "Generic (PLEG): container finished" podID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerID="b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157" exitCode=60
Oct 30 22:45:05.193485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.193422    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerDied","Data":"b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"}
Oct 30 22:45:05.193604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.193592    4410 scope.go:122] "RemoveContainer" containerID="b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"
Oct 30 22:45:05.193768 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:05.193753    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx_odh-ai-gateway-infra(d760d22b-efd0-490f-b4f1-6d91c7d5da85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85"
Oct 30 22:45:05.204405 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.204387    4410 scope.go:122] "RemoveContainer" containerID="a6a740fbf062fee4b82b0bb0906abeb21eebdfee76e47a9703768a14100fef42"
Oct 30 22:45:05.216292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.216272    4410 scope.go:122] "RemoveContainer" containerID="4d013eca726b1ab15cb0944a12dd4fe27bc303fde0192e418606d2feeffb91ae"
Oct 30 22:45:05.228658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.228634    4410 scope.go:122] "RemoveContainer" containerID="554bcd24ba179379d057f0c81d5fbfc8568b8adeb3aea295327f7cf134158a7d"
Oct 30 22:45:05.241363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.241348    4410 scope.go:122] "RemoveContainer" containerID="289cf42f9fe12cfd90a3345e21b93683c8c63d059d476f020d7d4dc7fa986020"
Oct 30 22:45:05.253095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.253078    4410 scope.go:122] "RemoveContainer" containerID="b7473cb65b366bd1ddad2be0147c94749d1887e9f651ede685f16a47c1e16a3c"
Oct 30 22:45:05.264898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:05.264882    4410 scope.go:122] "RemoveContainer" containerID="3ae14c8a5e03cf7d7798c37561036d9c0730f1557d685769d809c518a7710f03"
Oct 30 22:45:06.169620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.169581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:06.202978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.202951    4410 scope.go:122] "RemoveContainer" containerID="f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"
Oct 30 22:45:06.203401 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.203204    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq_odh-ai-gateway-infra(ad6de6f8-5a01-483c-abd5-4a1af9488ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7"
Oct 30 22:45:06.204634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.204610    4410 scope.go:122] "RemoveContainer" containerID="0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"
Oct 30 22:45:06.204900 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.204876    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75_odh-ai-gateway-infra(e915b9b8-f770-4dd8-bb14-1f5ca60aba2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e"
Oct 30 22:45:06.206187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.206167    4410 scope.go:122] "RemoveContainer" containerID="988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"
Oct 30 22:45:06.206414 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.206394    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn_odh-ai-gateway-infra(b77d9452-8280-41b9-b8e0-c702d516e15d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d"
Oct 30 22:45:06.207595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.207578    4410 scope.go:122] "RemoveContainer" containerID="b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"
Oct 30 22:45:06.207839 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.207819    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb_odh-ai-gateway-infra(75ce4291-9682-4b75-8079-a7502f4c5816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" podUID="75ce4291-9682-4b75-8079-a7502f4c5816"
Oct 30 22:45:06.209081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.209064    4410 scope.go:122] "RemoveContainer" containerID="b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"
Oct 30 22:45:06.209284 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.209265    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx_odh-ai-gateway-infra(d760d22b-efd0-490f-b4f1-6d91c7d5da85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85"
Oct 30 22:45:06.210599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.210581    4410 scope.go:122] "RemoveContainer" containerID="ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"
Oct 30 22:45:06.210838 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.210814    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658_odh-ai-gateway-infra(cdf67d3b-c9ed-4047-914d-cc1723471db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3"
Oct 30 22:45:06.212019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.212002    4410 scope.go:122] "RemoveContainer" containerID="de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"
Oct 30 22:45:06.212205 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.212186    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m_odh-ai-gateway-infra(659fe955-2d44-4057-8a77-974edb3ec5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9"
Oct 30 22:45:06.213460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.213443    4410 scope.go:122] "RemoveContainer" containerID="ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"
Oct 30 22:45:06.213648 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:06.213630    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh_odh-ai-gateway-infra(2b2940da-362c-48ec-a17c-36b85e65d321)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" podUID="2b2940da-362c-48ec-a17c-36b85e65d321"
Oct 30 22:45:06.707262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:06.707228    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:08.290505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:08.290462    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:08.896528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:08.896473    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:09.083743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:09.083690    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:11.895519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:11.895485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:12.053938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:12.053896    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"]
Oct 30 22:45:12.096903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:12.096869    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ffec2f-4964k"]
Oct 30 22:45:13.021003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:13.020970    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b2b61f-c7cf-4f8d-8269-5e3c52ddce84" path="/var/lib/kubelet/pods/21b2b61f-c7cf-4f8d-8269-5e3c52ddce84/volumes"
Oct 30 22:45:13.598570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:13.598531    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:13.830327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:13.830293    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:15.021140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:15.021104    4410 scope.go:122] "RemoveContainer" containerID="9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"
Oct 30 22:45:15.021521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:15.021197    4410 scope.go:122] "RemoveContainer" containerID="ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"
Oct 30 22:45:16.012392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.012363    4410 scope.go:122] "RemoveContainer" containerID="19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"
Oct 30 22:45:16.012527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.012451    4410 scope.go:122] "RemoveContainer" containerID="f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"
Oct 30 22:45:16.012719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.012699    4410 scope.go:122] "RemoveContainer" containerID="0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"
Oct 30 22:45:16.012819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.012768    4410 scope.go:122] "RemoveContainer" containerID="e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"
Oct 30 22:45:16.012897 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.012818    4410 scope.go:122] "RemoveContainer" containerID="988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"
Oct 30 22:45:16.299425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.299400    4410 generic.go:356] "Generic (PLEG): container finished" podID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerID="b8d3d76d53b291060ef07678a3a514ff6282732826171e8dc08fefee735ad894" exitCode=0
Oct 30 22:45:16.299845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.299468    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerDied","Data":"b8d3d76d53b291060ef07678a3a514ff6282732826171e8dc08fefee735ad894"}
Oct 30 22:45:16.299968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.299942    4410 scope.go:122] "RemoveContainer" containerID="b8d3d76d53b291060ef07678a3a514ff6282732826171e8dc08fefee735ad894"
Oct 30 22:45:16.301685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301658    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:16.301685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301682    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerDied","Data":"470cac982817d6804c39642a4cbe0a5d0e2ca942d5ec54466441c2d3afccfcb0"}
Oct 30 22:45:16.301873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301725    4410 scope.go:122] "RemoveContainer" containerID="9bdeb4b659680a33f46383d4f5a468f27bda89df6d828d139f96586240d5cde5"
Oct 30 22:45:16.301873 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301725    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:16.301986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301664    4410 generic.go:356] "Generic (PLEG): container finished" podID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerID="470cac982817d6804c39642a4cbe0a5d0e2ca942d5ec54466441c2d3afccfcb0" exitCode=60
Oct 30 22:45:16.302046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.301921    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:16.302104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.302060    4410 scope.go:122] "RemoveContainer" containerID="470cac982817d6804c39642a4cbe0a5d0e2ca942d5ec54466441c2d3afccfcb0"
Oct 30 22:45:16.302337 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:16.302311    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg_odh-ai-gateway-infra(2f02fb68-1152-4976-a9df-e607572f5e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b"
Oct 30 22:45:16.304140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.304117    4410 generic.go:356] "Generic (PLEG): container finished" podID="2b2940da-362c-48ec-a17c-36b85e65d321" containerID="48dc186beb6589e564792fb7af3f1076cac7c880b2c61ade187aebad702e1202" exitCode=60
Oct 30 22:45:16.304237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.304165    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerDied","Data":"48dc186beb6589e564792fb7af3f1076cac7c880b2c61ade187aebad702e1202"}
Oct 30 22:45:16.304437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.304425    4410 scope.go:122] "RemoveContainer" containerID="48dc186beb6589e564792fb7af3f1076cac7c880b2c61ade187aebad702e1202"
Oct 30 22:45:16.304658 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:16.304602    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh_odh-ai-gateway-infra(2b2940da-362c-48ec-a17c-36b85e65d321)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" podUID="2b2940da-362c-48ec-a17c-36b85e65d321"
Oct 30 22:45:16.346880 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:16.346855    4410 scope.go:122] "RemoveContainer" containerID="ca3cca0a4d05d6c460a08350f928a4477567ba10645241b3c3b69c99f6f72e9f"
Oct 30 22:45:17.022103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.022073    4410 scope.go:122] "RemoveContainer" containerID="de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"
Oct 30 22:45:17.022273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.022254    4410 scope.go:122] "RemoveContainer" containerID="b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"
Oct 30 22:45:17.204365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.204334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.314400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.314372    4410 generic.go:356] "Generic (PLEG): container finished" podID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerID="996b18e2a47849905400a9202314d7e16edcc66af32705285e1be27febc0e032" exitCode=60
Oct 30 22:45:17.314828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.314438    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerDied","Data":"996b18e2a47849905400a9202314d7e16edcc66af32705285e1be27febc0e032"}
Oct 30 22:45:17.314828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.314495    4410 scope.go:122] "RemoveContainer" containerID="f41c1988a33b900fff243891f515fceaddc5e2e59950c1edc4a9abbc8d181c93"
Oct 30 22:45:17.314957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.314912    4410 scope.go:122] "RemoveContainer" containerID="996b18e2a47849905400a9202314d7e16edcc66af32705285e1be27febc0e032"
Oct 30 22:45:17.315789 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:17.315748    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq_odh-ai-gateway-infra(ad6de6f8-5a01-483c-abd5-4a1af9488ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7"
Oct 30 22:45:17.316914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.316889    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerStarted","Data":"e9b449618d784a38120a883732be24067ea3639c6550da8cf61c89fdac256a4e"}
Oct 30 22:45:17.318917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.318897    4410 generic.go:356] "Generic (PLEG): container finished" podID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerID="6deedd5d3c6533a0b918f52443208541ee1960f2e075d6ae63b2091718c80c86" exitCode=60
Oct 30 22:45:17.319018 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.318917    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerDied","Data":"6deedd5d3c6533a0b918f52443208541ee1960f2e075d6ae63b2091718c80c86"}
Oct 30 22:45:17.319244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.319228    4410 scope.go:122] "RemoveContainer" containerID="6deedd5d3c6533a0b918f52443208541ee1960f2e075d6ae63b2091718c80c86"
Oct 30 22:45:17.319535 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:17.319515    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp_odh-ai-gateway-infra(085629ab-4bfd-461e-beac-0ad6a2cacbb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6"
Oct 30 22:45:17.321010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.320986    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerStarted","Data":"8c8b32a3239d31c48a80a71994bbdca3ccd010de2407178b2f88cd82b4706bad"}
Oct 30 22:45:17.322992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.322971    4410 generic.go:356] "Generic (PLEG): container finished" podID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerID="e43ed52cfe8e418cb01d1de0db06e30b8d646ec9b5f1b44d394ae4cc2b99ba1a" exitCode=60
Oct 30 22:45:17.323109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.323041    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerDied","Data":"e43ed52cfe8e418cb01d1de0db06e30b8d646ec9b5f1b44d394ae4cc2b99ba1a"}
Oct 30 22:45:17.323404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.323382    4410 scope.go:122] "RemoveContainer" containerID="e43ed52cfe8e418cb01d1de0db06e30b8d646ec9b5f1b44d394ae4cc2b99ba1a"
Oct 30 22:45:17.323622 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:17.323600    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn_odh-ai-gateway-infra(b77d9452-8280-41b9-b8e0-c702d516e15d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d"
Oct 30 22:45:17.326740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.326722    4410 generic.go:356] "Generic (PLEG): container finished" podID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerID="f2287c3a9e72ab7c997791d3948f742203d220fbe99328da2437c31e28160b69" exitCode=60
Oct 30 22:45:17.326848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.326764    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerDied","Data":"f2287c3a9e72ab7c997791d3948f742203d220fbe99328da2437c31e28160b69"}
Oct 30 22:45:17.327060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.327047    4410 scope.go:122] "RemoveContainer" containerID="f2287c3a9e72ab7c997791d3948f742203d220fbe99328da2437c31e28160b69"
Oct 30 22:45:17.327291 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:17.327271    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x_odh-ai-gateway-infra(b16dda5b-a4dd-45b9-941d-45d09d307557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557"
Oct 30 22:45:17.385144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.385119    4410 scope.go:122] "RemoveContainer" containerID="19154aae956d56040a30a3097599bdc56eff3c4902569a45ecf50e46554ac694"
Oct 30 22:45:17.414399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.414314    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.428811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.428756    4410 scope.go:122] "RemoveContainer" containerID="988542499b4ada70eee39429960cebc2d4beae8bc70da8e818d140c2aacfbad7"
Oct 30 22:45:17.442918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.442887    4410 scope.go:122] "RemoveContainer" containerID="e3c5f0497a1187d7ba8fed15b79855f3a1fa79ee49efd6fc3080f4e7cda67dd6"
Oct 30 22:45:17.607864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.606513    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.609839 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.609809    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:17.713754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.713721    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:17.774821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.774758    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.828222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.828195    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:17.915054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.915019    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\" (UniqueName: \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\") pod \"2f02fb68-1152-4976-a9df-e607572f5e3b\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") "
Oct 30 22:45:17.915200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.915108    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\") pod \"2f02fb68-1152-4976-a9df-e607572f5e3b\" (UID: \"2f02fb68-1152-4976-a9df-e607572f5e3b\") "
Oct 30 22:45:17.915510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.915432    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca" pod "2f02fb68-1152-4976-a9df-e607572f5e3b" (UID: "2f02fb68-1152-4976-a9df-e607572f5e3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.915786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.915751    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f02fb68-1152-4976-a9df-e607572f5e3b-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.917890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.917857    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl" pod "2f02fb68-1152-4976-a9df-e607572f5e3b" (UID: "2f02fb68-1152-4976-a9df-e607572f5e3b"). InnerVolumeSpecName "kube-api-access-5lrjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.940730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:17.940707    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:18.012741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.012702    4410 scope.go:122] "RemoveContainer" containerID="b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"
Oct 30 22:45:18.013946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.013923    4410 scope.go:122] "RemoveContainer" containerID="ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"
Oct 30 22:45:18.016668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.016639    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\" (UniqueName: \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\") pod \"2b2940da-362c-48ec-a17c-36b85e65d321\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") "
Oct 30 22:45:18.029069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.017891    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\") pod \"2b2940da-362c-48ec-a17c-36b85e65d321\" (UID: \"2b2940da-362c-48ec-a17c-36b85e65d321\") "
Oct 30 22:45:18.029069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.018282    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrjl\" (UniqueName: \"kubernetes.io/projected/2f02fb68-1152-4976-a9df-e607572f5e3b-kube-api-access-5lrjl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.029069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.018786    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca" pod "2b2940da-362c-48ec-a17c-36b85e65d321" (UID: "2b2940da-362c-48ec-a17c-36b85e65d321"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.029069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.022731    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz" pod "2b2940da-362c-48ec-a17c-36b85e65d321" (UID: "2b2940da-362c-48ec-a17c-36b85e65d321"). InnerVolumeSpecName "kube-api-access-vp8jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.073005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.064725    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:45:18.073005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.064855    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:45:18.073005 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.064894    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:45:18.119816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.119789    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2940da-362c-48ec-a17c-36b85e65d321-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.119927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.119816    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8jz\" (UniqueName: \"kubernetes.io/projected/2b2940da-362c-48ec-a17c-36b85e65d321-kube-api-access-vp8jz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.339014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.338988    4410 generic.go:356] "Generic (PLEG): container finished" podID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerID="8c8b32a3239d31c48a80a71994bbdca3ccd010de2407178b2f88cd82b4706bad" exitCode=60
Oct 30 22:45:18.339369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.339056    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerDied","Data":"8c8b32a3239d31c48a80a71994bbdca3ccd010de2407178b2f88cd82b4706bad"}
Oct 30 22:45:18.339369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.339119    4410 scope.go:122] "RemoveContainer" containerID="0286933e308765c9d62bee208ef0b34542f53a388a592bb20d0e2972bed5bff2"
Oct 30 22:45:18.339496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.339481    4410 scope.go:122] "RemoveContainer" containerID="8c8b32a3239d31c48a80a71994bbdca3ccd010de2407178b2f88cd82b4706bad"
Oct 30 22:45:18.339847 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:18.339822    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75_odh-ai-gateway-infra(e915b9b8-f770-4dd8-bb14-1f5ca60aba2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e"
Oct 30 22:45:18.342390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.342370    4410 generic.go:356] "Generic (PLEG): container finished" podID="75ce4291-9682-4b75-8079-a7502f4c5816" containerID="848b304cf729ddf1fe854620b9aa30dcf3dd52b7c1f75b3a73e2d82abfcebb56" exitCode=60
Oct 30 22:45:18.342486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.342436    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerDied","Data":"848b304cf729ddf1fe854620b9aa30dcf3dd52b7c1f75b3a73e2d82abfcebb56"}
Oct 30 22:45:18.342853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.342831    4410 scope.go:122] "RemoveContainer" containerID="848b304cf729ddf1fe854620b9aa30dcf3dd52b7c1f75b3a73e2d82abfcebb56"
Oct 30 22:45:18.343128 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:18.343104    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb_odh-ai-gateway-infra(75ce4291-9682-4b75-8079-a7502f4c5816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" podUID="75ce4291-9682-4b75-8079-a7502f4c5816"
Oct 30 22:45:18.343907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.343884    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg" event={"ID":"2f02fb68-1152-4976-a9df-e607572f5e3b","Type":"ContainerDied","Data":"79fdf81cacfcf6fe89324945ea1beb9d057a353335b08392e26c158582202c74"}
Oct 30 22:45:18.343907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.343894    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"
Oct 30 22:45:18.372950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.372917    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerStarted","Data":"486c81f0bc563df458a37876eb66918affeb45a8af6c3e945f3e5799bb37797c"}
Oct 30 22:45:18.374610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.374580    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh" event={"ID":"2b2940da-362c-48ec-a17c-36b85e65d321","Type":"ContainerDied","Data":"98d2089726a78ddff16a279222f857d045459c64f0b98034fe3bdd237973b544"}
Oct 30 22:45:18.374749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.374596    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"
Oct 30 22:45:18.476798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.475864    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:18.482487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.482461    4410 scope.go:122] "RemoveContainer" containerID="b5a6f09b1eec6c941ee92b4d4fa9c7da3b4295a549910410cbe0a6c8ba2d8dd1"
Oct 30 22:45:18.501118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.501090    4410 scope.go:122] "RemoveContainer" containerID="470cac982817d6804c39642a4cbe0a5d0e2ca942d5ec54466441c2d3afccfcb0"
Oct 30 22:45:18.513864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.513848    4410 scope.go:122] "RemoveContainer" containerID="48dc186beb6589e564792fb7af3f1076cac7c880b2c61ade187aebad702e1202"
Oct 30 22:45:18.582420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.582232    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:18.695202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.694259    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:18.700663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.699725    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:18.705092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.704835    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b8717e-29890005-8bkmh"]
Oct 30 22:45:18.707833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.707806    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:18.733249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.733214    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\" (UniqueName: \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\") pod \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") "
Oct 30 22:45:18.733605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.733579    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\") pod \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\" (UID: \"ad6de6f8-5a01-483c-abd5-4a1af9488ee7\") "
Oct 30 22:45:18.733993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.733968    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca" pod "ad6de6f8-5a01-483c-abd5-4a1af9488ee7" (UID: "ad6de6f8-5a01-483c-abd5-4a1af9488ee7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.734301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.734280    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.736698 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.736676    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm" pod "ad6de6f8-5a01-483c-abd5-4a1af9488ee7" (UID: "ad6de6f8-5a01-483c-abd5-4a1af9488ee7"). InnerVolumeSpecName "kube-api-access-r64tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.787658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.787635    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:18.823633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.823598    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:18.839321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.838896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\") pod \"b16dda5b-a4dd-45b9-941d-45d09d307557\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") "
Oct 30 22:45:18.839321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.839081    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\" (UniqueName: \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\") pod \"b16dda5b-a4dd-45b9-941d-45d09d307557\" (UID: \"b16dda5b-a4dd-45b9-941d-45d09d307557\") "
Oct 30 22:45:18.840251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.839927    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca" pod "b16dda5b-a4dd-45b9-941d-45d09d307557" (UID: "b16dda5b-a4dd-45b9-941d-45d09d307557"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.840764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.840714    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16dda5b-a4dd-45b9-941d-45d09d307557-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.840764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.840744    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r64tm\" (UniqueName: \"kubernetes.io/projected/ad6de6f8-5a01-483c-abd5-4a1af9488ee7-kube-api-access-r64tm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.844554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.844520    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7" pod "b16dda5b-a4dd-45b9-941d-45d09d307557" (UID: "b16dda5b-a4dd-45b9-941d-45d09d307557"). InnerVolumeSpecName "kube-api-access-g75z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.888794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.888757    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:18.938966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.938936    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:18.941416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.941392    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\" (UniqueName: \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\") pod \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") "
Oct 30 22:45:18.941590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.941574    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\") pod \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\" (UID: \"085629ab-4bfd-461e-beac-0ad6a2cacbb6\") "
Oct 30 22:45:18.942067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.942044    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75z7\" (UniqueName: \"kubernetes.io/projected/b16dda5b-a4dd-45b9-941d-45d09d307557-kube-api-access-g75z7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.942163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.942106    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca" pod "085629ab-4bfd-461e-beac-0ad6a2cacbb6" (UID: "085629ab-4bfd-461e-beac-0ad6a2cacbb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.943999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.943973    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl" pod "085629ab-4bfd-461e-beac-0ad6a2cacbb6" (UID: "085629ab-4bfd-461e-beac-0ad6a2cacbb6"). InnerVolumeSpecName "kube-api-access-jmxxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.944445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.944423    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a0116e-29890005-rh5vg"]
Oct 30 22:45:18.979881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:18.979843    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:19.022631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.022606    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:19.027614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.027545    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" path="/var/lib/kubelet/pods/2b2940da-362c-48ec-a17c-36b85e65d321/volumes"
Oct 30 22:45:19.028211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.028191    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" path="/var/lib/kubelet/pods/2f02fb68-1152-4976-a9df-e607572f5e3b/volumes"
Oct 30 22:45:19.042999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.042977    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/085629ab-4bfd-461e-beac-0ad6a2cacbb6-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.043159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.043141    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxxl\" (UniqueName: \"kubernetes.io/projected/085629ab-4bfd-461e-beac-0ad6a2cacbb6-kube-api-access-jmxxl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.080017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.079156    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:45:19.080017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.079207    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:45:19.143605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.143569    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\") pod \"b77d9452-8280-41b9-b8e0-c702d516e15d\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") "
Oct 30 22:45:19.143725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.143677    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\" (UniqueName: \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\") pod \"b77d9452-8280-41b9-b8e0-c702d516e15d\" (UID: \"b77d9452-8280-41b9-b8e0-c702d516e15d\") "
Oct 30 22:45:19.143901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.143879    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca" pod "b77d9452-8280-41b9-b8e0-c702d516e15d" (UID: "b77d9452-8280-41b9-b8e0-c702d516e15d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.144589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.144567    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77d9452-8280-41b9-b8e0-c702d516e15d-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.174961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.174923    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb" pod "b77d9452-8280-41b9-b8e0-c702d516e15d" (UID: "b77d9452-8280-41b9-b8e0-c702d516e15d"). InnerVolumeSpecName "kube-api-access-8qmgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.269805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.245455    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmgb\" (UniqueName: \"kubernetes.io/projected/b77d9452-8280-41b9-b8e0-c702d516e15d-kube-api-access-8qmgb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.383581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.383499    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x" event={"ID":"b16dda5b-a4dd-45b9-941d-45d09d307557","Type":"ContainerDied","Data":"37db58c4c18dcc965ffdf0d330efff2fb74b2b9690eca0c653e72852c23f7653"}
Oct 30 22:45:19.384097 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.383541    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"
Oct 30 22:45:19.384326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.384076    4410 scope.go:122] "RemoveContainer" containerID="f2287c3a9e72ab7c997791d3948f742203d220fbe99328da2437c31e28160b69"
Oct 30 22:45:19.385857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.385831    4410 generic.go:356] "Generic (PLEG): container finished" podID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerID="4578db041ea694f33d95db6caeaa5a1c85bf735af626780c768a6946d064f31c" exitCode=60
Oct 30 22:45:19.385968 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.385943    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerDied","Data":"4578db041ea694f33d95db6caeaa5a1c85bf735af626780c768a6946d064f31c"}
Oct 30 22:45:19.386402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.386381    4410 scope.go:122] "RemoveContainer" containerID="4578db041ea694f33d95db6caeaa5a1c85bf735af626780c768a6946d064f31c"
Oct 30 22:45:19.386690 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:19.386663    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658_odh-ai-gateway-infra(cdf67d3b-c9ed-4047-914d-cc1723471db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3"
Oct 30 22:45:19.388265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.388242    4410 generic.go:356] "Generic (PLEG): container finished" podID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerID="486c81f0bc563df458a37876eb66918affeb45a8af6c3e945f3e5799bb37797c" exitCode=60
Oct 30 22:45:19.388383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.388327    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerDied","Data":"486c81f0bc563df458a37876eb66918affeb45a8af6c3e945f3e5799bb37797c"}
Oct 30 22:45:19.388657 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.388643    4410 scope.go:122] "RemoveContainer" containerID="486c81f0bc563df458a37876eb66918affeb45a8af6c3e945f3e5799bb37797c"
Oct 30 22:45:19.388877 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:19.388859    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m_odh-ai-gateway-infra(659fe955-2d44-4057-8a77-974edb3ec5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9"
Oct 30 22:45:19.394739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.394721    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"
Oct 30 22:45:19.394969 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.394759    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq" event={"ID":"ad6de6f8-5a01-483c-abd5-4a1af9488ee7","Type":"ContainerDied","Data":"c35c3847acb3fb55a9e9b9498940ce8fa20b87420c4ddae7e86b745602a8d209"}
Oct 30 22:45:19.396447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.396430    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"
Oct 30 22:45:19.396686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.396441    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp" event={"ID":"085629ab-4bfd-461e-beac-0ad6a2cacbb6","Type":"ContainerDied","Data":"961e156a6f7c48450277d1890d58cb514d596502a212938e403208555cefc165"}
Oct 30 22:45:19.397376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.397357    4410 scope.go:122] "RemoveContainer" containerID="ee122e2b9d76a7f1b68de67574ce391e83f1c0da982ae01752dbd94cc217ecb3"
Oct 30 22:45:19.400393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.400375    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"
Oct 30 22:45:19.400481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.400391    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn" event={"ID":"b77d9452-8280-41b9-b8e0-c702d516e15d","Type":"ContainerDied","Data":"7f93ccf17694b41c48f15a5575bcb21dc6c0710b1203deac16f0fd8f5c29bdfc"}
Oct 30 22:45:19.404472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.404446    4410 generic.go:356] "Generic (PLEG): container finished" podID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerID="9ee72b23f4719bb2bd0f5f4af23819c0e1f930ee5938877c859d0bbae8a564ea" exitCode=60
Oct 30 22:45:19.404591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.404524    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerDied","Data":"9ee72b23f4719bb2bd0f5f4af23819c0e1f930ee5938877c859d0bbae8a564ea"}
Oct 30 22:45:19.404934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.404900    4410 scope.go:122] "RemoveContainer" containerID="9ee72b23f4719bb2bd0f5f4af23819c0e1f930ee5938877c859d0bbae8a564ea"
Oct 30 22:45:19.405170 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:19.405147    4410 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx_odh-ai-gateway-infra(d760d22b-efd0-490f-b4f1-6d91c7d5da85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85"
Oct 30 22:45:19.421330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.421304    4410 scope.go:122] "RemoveContainer" containerID="de94e7b1691f55844b00cdcf21f5868518be88ab627873299c234f31f476223a"
Oct 30 22:45:19.437907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.437882    4410 scope.go:122] "RemoveContainer" containerID="996b18e2a47849905400a9202314d7e16edcc66af32705285e1be27febc0e032"
Oct 30 22:45:19.473658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.473561    4410 scope.go:122] "RemoveContainer" containerID="6deedd5d3c6533a0b918f52443208541ee1960f2e075d6ae63b2091718c80c86"
Oct 30 22:45:19.485202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.485179    4410 scope.go:122] "RemoveContainer" containerID="e43ed52cfe8e418cb01d1de0db06e30b8d646ec9b5f1b44d394ae4cc2b99ba1a"
Oct 30 22:45:19.501740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.501719    4410 scope.go:122] "RemoveContainer" containerID="b1d5346f5f270e1af3027d1ce4fada2a82f5c7a02f0ca173eca15f22a3c3f157"
Oct 30 22:45:19.599038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.599010    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:19.605558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.605537    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:19.779819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.779378    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\" (UniqueName: \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\") pod \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") "
Oct 30 22:45:19.779819 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.779532    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\") pod \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\" (UID: \"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e\") "
Oct 30 22:45:19.780120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.779905    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca" pod "e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" (UID: "e915b9b8-f770-4dd8-bb14-1f5ca60aba2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.780120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.780029    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.782253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.782226    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv" pod "e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" (UID: "e915b9b8-f770-4dd8-bb14-1f5ca60aba2e"). InnerVolumeSpecName "kube-api-access-l25jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.825216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.825187    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:19.881208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.881176    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l25jv\" (UniqueName: \"kubernetes.io/projected/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e-kube-api-access-l25jv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.906479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.906453    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:19.911303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:19.911276    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e90906-29890005-kzx9x"]
Oct 30 22:45:20.018514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.018490    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:20.031679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.031620    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:20.035512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.035488    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89cf28-29890005-4vhbp"]
Oct 30 22:45:20.070320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.070205    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:45:20.070320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.070259    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:45:20.142877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.142842    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:20.171411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.171375    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20ccef-29890005-d8dgq"]
Oct 30 22:45:20.184152 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.184121    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\" (UniqueName: \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\") pod \"75ce4291-9682-4b75-8079-a7502f4c5816\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") "
Oct 30 22:45:20.184268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.184245    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\") pod \"75ce4291-9682-4b75-8079-a7502f4c5816\" (UID: \"75ce4291-9682-4b75-8079-a7502f4c5816\") "
Oct 30 22:45:20.184613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.184591    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca" pod "75ce4291-9682-4b75-8079-a7502f4c5816" (UID: "75ce4291-9682-4b75-8079-a7502f4c5816"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.184812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.184797    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ce4291-9682-4b75-8079-a7502f4c5816-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.186253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.186228    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw" pod "75ce4291-9682-4b75-8079-a7502f4c5816" (UID: "75ce4291-9682-4b75-8079-a7502f4c5816"). InnerVolumeSpecName "kube-api-access-schdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.286377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.286294    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schdw\" (UniqueName: \"kubernetes.io/projected/75ce4291-9682-4b75-8079-a7502f4c5816-kube-api-access-schdw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.331882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.331842    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:20.335250 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.335224    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4deef-29890005-448gn"]
Oct 30 22:45:20.419216 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.419195    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"
Oct 30 22:45:20.419611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.419193    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75" event={"ID":"e915b9b8-f770-4dd8-bb14-1f5ca60aba2e","Type":"ContainerDied","Data":"4f4349d668c12eb7c843c1d97c5e3154d2988d8ac116b3fd19998dce7be04445"}
Oct 30 22:45:20.419611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.419319    4410 scope.go:122] "RemoveContainer" containerID="8c8b32a3239d31c48a80a71994bbdca3ccd010de2407178b2f88cd82b4706bad"
Oct 30 22:45:20.421605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.421587    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"
Oct 30 22:45:20.421714 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.421613    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb" event={"ID":"75ce4291-9682-4b75-8079-a7502f4c5816","Type":"ContainerDied","Data":"e19fd0f244a5dcf6b7301ae240fdfc3f836f06ce9cbb182239597697f79a2094"}
Oct 30 22:45:20.431543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.431521    4410 scope.go:122] "RemoveContainer" containerID="848b304cf729ddf1fe854620b9aa30dcf3dd52b7c1f75b3a73e2d82abfcebb56"
Oct 30 22:45:20.531840 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:20.531800    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:45:20.611644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.611619    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:20.636663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.636385    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:20.677946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.677675    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e90e35-29890005-7bbrb"]
Oct 30 22:45:20.680817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.680755    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:20.792223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.792187    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\") pod \"659fe955-2d44-4057-8a77-974edb3ec5b9\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") "
Oct 30 22:45:20.792381 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.792361    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\" (UniqueName: \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\") pod \"659fe955-2d44-4057-8a77-974edb3ec5b9\" (UID: \"659fe955-2d44-4057-8a77-974edb3ec5b9\") "
Oct 30 22:45:20.792822 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.792794    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca" pod "659fe955-2d44-4057-8a77-974edb3ec5b9" (UID: "659fe955-2d44-4057-8a77-974edb3ec5b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.792957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.792922    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659fe955-2d44-4057-8a77-974edb3ec5b9-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.795360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.795333    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d" pod "659fe955-2d44-4057-8a77-974edb3ec5b9" (UID: "659fe955-2d44-4057-8a77-974edb3ec5b9"). InnerVolumeSpecName "kube-api-access-58s9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.805446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.805420    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:20.842343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.842259    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:20.875806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.874625    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d2a3e0-29890005-pdv75"]
Oct 30 22:45:20.881912 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.881887    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:20.894624 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.894596    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s9d\" (UniqueName: \"kubernetes.io/projected/659fe955-2d44-4057-8a77-974edb3ec5b9-kube-api-access-58s9d\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.963063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.963032    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:20.995302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.995265    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\") pod \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") "
Oct 30 22:45:20.995486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.995320    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\" (UniqueName: \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\") pod \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\" (UID: \"d760d22b-efd0-490f-b4f1-6d91c7d5da85\") "
Oct 30 22:45:20.995692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.995667    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca" pod "d760d22b-efd0-490f-b4f1-6d91c7d5da85" (UID: "d760d22b-efd0-490f-b4f1-6d91c7d5da85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.995849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.995830    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d760d22b-efd0-490f-b4f1-6d91c7d5da85-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.997435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:20.997411    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr" pod "d760d22b-efd0-490f-b4f1-6d91c7d5da85" (UID: "d760d22b-efd0-490f-b4f1-6d91c7d5da85"). InnerVolumeSpecName "kube-api-access-nksrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.020784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.020752    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" path="/var/lib/kubelet/pods/085629ab-4bfd-461e-beac-0ad6a2cacbb6/volumes"
Oct 30 22:45:21.021171 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.021156    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" path="/var/lib/kubelet/pods/75ce4291-9682-4b75-8079-a7502f4c5816/volumes"
Oct 30 22:45:21.021509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.021496    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" path="/var/lib/kubelet/pods/ad6de6f8-5a01-483c-abd5-4a1af9488ee7/volumes"
Oct 30 22:45:21.022372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.022353    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" path="/var/lib/kubelet/pods/b16dda5b-a4dd-45b9-941d-45d09d307557/volumes"
Oct 30 22:45:21.022853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.022838    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" path="/var/lib/kubelet/pods/b77d9452-8280-41b9-b8e0-c702d516e15d/volumes"
Oct 30 22:45:21.023308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.023291    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" path="/var/lib/kubelet/pods/e915b9b8-f770-4dd8-bb14-1f5ca60aba2e/volumes"
Oct 30 22:45:21.070053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.069188    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:45:21.070053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.069875    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:45:21.097045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.096984    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\") pod \"cdf67d3b-c9ed-4047-914d-cc1723471db3\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") "
Oct 30 22:45:21.097153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.097102    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\" (UniqueName: \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\") pod \"cdf67d3b-c9ed-4047-914d-cc1723471db3\" (UID: \"cdf67d3b-c9ed-4047-914d-cc1723471db3\") "
Oct 30 22:45:21.097295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.097274    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca" pod "cdf67d3b-c9ed-4047-914d-cc1723471db3" (UID: "cdf67d3b-c9ed-4047-914d-cc1723471db3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.097613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.097578    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf67d3b-c9ed-4047-914d-cc1723471db3-maas-api-service-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.097613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.097606    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nksrr\" (UniqueName: \"kubernetes.io/projected/d760d22b-efd0-490f-b4f1-6d91c7d5da85-kube-api-access-nksrr\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.099120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.099096    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt" pod "cdf67d3b-c9ed-4047-914d-cc1723471db3" (UID: "cdf67d3b-c9ed-4047-914d-cc1723471db3"). InnerVolumeSpecName "kube-api-access-n2crt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.198645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.198612    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2crt\" (UniqueName: \"kubernetes.io/projected/cdf67d3b-c9ed-4047-914d-cc1723471db3-kube-api-access-n2crt\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.433934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.433906    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"
Oct 30 22:45:21.434385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.433937    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx" event={"ID":"d760d22b-efd0-490f-b4f1-6d91c7d5da85","Type":"ContainerDied","Data":"fd6e33b95e8c5b0d526b365c5e68b959040ede0263a6852e9621695952dcfc7e"}
Oct 30 22:45:21.434385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.434001    4410 scope.go:122] "RemoveContainer" containerID="9ee72b23f4719bb2bd0f5f4af23819c0e1f930ee5938877c859d0bbae8a564ea"
Oct 30 22:45:21.435393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.435369    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658" event={"ID":"cdf67d3b-c9ed-4047-914d-cc1723471db3","Type":"ContainerDied","Data":"ad7225f7450b009cdd10fb2423ba41286dc25b6197d1facf1d8cfdfbeb7cb30b"}
Oct 30 22:45:21.435466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.435377    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"
Oct 30 22:45:21.437219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.437198    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m" event={"ID":"659fe955-2d44-4057-8a77-974edb3ec5b9","Type":"ContainerDied","Data":"dcc4e4754b04cd971c0cb14805fa363c14a245bc45f87dc1e6efd63fb8be796a"}
Oct 30 22:45:21.437294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.437252    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"
Oct 30 22:45:21.444432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.444416    4410 scope.go:122] "RemoveContainer" containerID="4578db041ea694f33d95db6caeaa5a1c85bf735af626780c768a6946d064f31c"
Oct 30 22:45:21.477352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.477330    4410 scope.go:122] "RemoveContainer" containerID="486c81f0bc563df458a37876eb66918affeb45a8af6c3e945f3e5799bb37797c"
Oct 30 22:45:21.600282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.600247    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:21.609531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.609503    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c788d-29890005-6fn2m"]
Oct 30 22:45:21.727011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.726931    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:21.737137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.737112    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-15e0f6-29890005-pclgx"]
Oct 30 22:45:21.863004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.862968    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:21.868802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:21.867579    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90e0d5-29890005-pl658"]
Oct 30 22:45:22.066913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:22.066828    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:45:22.067173 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:22.067147    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:45:22.392032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:22.392004    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:23.021788 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.021731    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" path="/var/lib/kubelet/pods/659fe955-2d44-4057-8a77-974edb3ec5b9/volumes"
Oct 30 22:45:23.022326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.022308    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" path="/var/lib/kubelet/pods/cdf67d3b-c9ed-4047-914d-cc1723471db3/volumes"
Oct 30 22:45:23.022818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.022801    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" path="/var/lib/kubelet/pods/d760d22b-efd0-490f-b4f1-6d91c7d5da85/volumes"
Oct 30 22:45:23.048634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.048608    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"]
Oct 30 22:45:23.051842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.051821    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8175a5-8f2b5"]
Oct 30 22:45:23.064214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.064184    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:45:23.064493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.064474    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:45:23.065264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:23.065245    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:45:24.910313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:24.910274    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:25.020134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:25.020101    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a8d7df-1138-41cc-a9c5-59182d6aac93" path="/var/lib/kubelet/pods/c3a8d7df-1138-41cc-a9c5-59182d6aac93/volumes"
Oct 30 22:45:27.210662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:27.210622    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:29.537669 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:29.537634    4410 scope.go:122] "RemoveContainer" containerID="490581ae856e279bff0fcedfc9d3d46f5d25c16fe4ce9e5709f0ae2fddb221e2"
Oct 30 22:45:32.080934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:32.080905    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.212542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:35.212504    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.366302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:35.366267    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:37.030702 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:37.030668    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:38.407471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:38.407437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.060847 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:39.060808    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.271682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:39.271645    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.820517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:39.820461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:41.981263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:41.981222    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.260458 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:43.260428    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.437606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:43.437576    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.604440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:43.604351    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.518301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.518263    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.704219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.704184    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.920262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920211    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:45:46.920763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920748    4410 cpu_manager.go:414] "containerMap: removing container" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920766    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920794    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920801    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920812    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920817    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920824    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920829    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920838    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.920850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920845    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920872    4410 cpu_manager.go:414] "containerMap: removing container" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920878    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920884    4410 cpu_manager.go:414] "containerMap: removing container" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920889    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920901    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920906    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920912    4410 cpu_manager.go:414] "containerMap: removing container" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920917    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920928    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920932    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920940    4410 cpu_manager.go:414] "containerMap: removing container" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920947    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920956    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920961    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920972    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920976    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920985    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920989    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.920998    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921002    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921008    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921012    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921019    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921023    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921034    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921039    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921045    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921049    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921055    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921059    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921068    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921072    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921081    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921087    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921097    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921101    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921107    4410 cpu_manager.go:414] "containerMap: removing container" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.921144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921111    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921191    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921199    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921207    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921212    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921217    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921223    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921228    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921234    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921240    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921246    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921251    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921256    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921268    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921274    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921280    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921285    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921290    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d760d22b-efd0-490f-b4f1-6d91c7d5da85" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921297    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921306    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921313    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2940da-362c-48ec-a17c-36b85e65d321" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921319    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921325    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921330    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921335    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921339    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921345    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921349    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:46.922317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.921355    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:46.924639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.924622    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:46.975720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.975687    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:46.975891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.975744    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt94b\" (UniqueName: \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:46.981724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:46.981691    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:45:47.030197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.030165    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-32af64\""
Oct 30 22:45:47.030489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.030465    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-32af64-dockercfg-9rqgd\""
Oct 30 22:45:47.077179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.077144    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:47.077471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.077439    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt94b\" (UniqueName: \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:47.078577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.078552    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:47.082629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.082609    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:45:47.083237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083220    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083238    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083250    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083255    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e915b9b8-f770-4dd8-bb14-1f5ca60aba2e" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083268    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083276    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083290    4410 cpu_manager.go:414] "containerMap: removing container" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083295    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083304    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083308    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:47.083316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083318    4410 cpu_manager.go:414] "containerMap: removing container" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083323    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ce4291-9682-4b75-8079-a7502f4c5816" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083336    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083341    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083348    4410 cpu_manager.go:414] "containerMap: removing container" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083354    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085629ab-4bfd-461e-beac-0ad6a2cacbb6" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083373    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083378    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf67d3b-c9ed-4047-914d-cc1723471db3" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083463    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16dda5b-a4dd-45b9-941d-45d09d307557" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083475    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="659fe955-2d44-4057-8a77-974edb3ec5b9" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083484    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6de6f8-5a01-483c-abd5-4a1af9488ee7" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083490    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f02fb68-1152-4976-a9df-e607572f5e3b" containerName="cleanup"
Oct 30 22:45:47.083694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.083496    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77d9452-8280-41b9-b8e0-c702d516e15d" containerName="cleanup"
Oct 30 22:45:47.087079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.087063    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.135196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.135166    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt94b\" (UniqueName: \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\") pod \"payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:47.145354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.145326    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:45:47.178568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.178495    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8whz\" (UniqueName: \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.178696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.178572    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.237672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.237641    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:47.280335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.280294    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8whz\" (UniqueName: \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.280527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.280370    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.281133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.281112    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.339493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.339459    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8whz\" (UniqueName: \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\") pod \"payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.391685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.391657    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:47.400632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.400603    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:47.611887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.611861    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:45:47.613357 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:47.613330    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d7cfd1_cb7c_452f_876f_2fbc6ffaa306.slice/crio-4fd19e36f3e184b820727412aa0a41677d20dfdc76f370c0727b2b3e0d7f0476 WatchSource:0}: Error finding container 4fd19e36f3e184b820727412aa0a41677d20dfdc76f370c0727b2b3e0d7f0476: Status 404 returned error can't find the container with id 4fd19e36f3e184b820727412aa0a41677d20dfdc76f370c0727b2b3e0d7f0476
Oct 30 22:45:47.646688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.646665    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" event={"ID":"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306","Type":"ContainerStarted","Data":"4fd19e36f3e184b820727412aa0a41677d20dfdc76f370c0727b2b3e0d7f0476"}
Oct 30 22:45:47.786695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:47.786666    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:45:47.788147 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:45:47.788120    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7891b92e_6d14_4644_ba7e_f2123858d752.slice/crio-0730ffcc8d4f2483474b72f961ca3401333ad847cbe20e26872cc58bae316b1b WatchSource:0}: Error finding container 0730ffcc8d4f2483474b72f961ca3401333ad847cbe20e26872cc58bae316b1b: Status 404 returned error can't find the container with id 0730ffcc8d4f2483474b72f961ca3401333ad847cbe20e26872cc58bae316b1b
Oct 30 22:45:48.653366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.653331    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" event={"ID":"7891b92e-6d14-4644-ba7e-f2123858d752","Type":"ContainerStarted","Data":"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"}
Oct 30 22:45:48.653934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.653381    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" event={"ID":"7891b92e-6d14-4644-ba7e-f2123858d752","Type":"ContainerStarted","Data":"0730ffcc8d4f2483474b72f961ca3401333ad847cbe20e26872cc58bae316b1b"}
Oct 30 22:45:48.653934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.653440    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:48.655075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.655045    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" event={"ID":"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306","Type":"ContainerStarted","Data":"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"}
Oct 30 22:45:48.655194 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.655091    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:45:48.777195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.777142    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" podStartSLOduration=1.7771284710000002 podStartE2EDuration="1.777128471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:48.774020256 +0000 UTC m=+3801.862758060" watchObservedRunningTime="2026-10-30 22:45:48.777128471 +0000 UTC m=+3801.865866274"
Oct 30 22:45:48.895797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:48.894957    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" podStartSLOduration=2.894893693 podStartE2EDuration="2.894893693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:48.890890694 +0000 UTC m=+3801.979628501" watchObservedRunningTime="2026-10-30 22:45:48.894893693 +0000 UTC m=+3801.983631513"
Oct 30 22:45:50.401585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:50.401553    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:50.522263 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:45:50.522229    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:45:51.366521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:51.366484    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:51.593660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:51.593624    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.328425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:52.328387    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.487824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:52.487765    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.657518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:52.657464    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:57.547145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:57.547100    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:59.664534 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:59.664499    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:45:59.664932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:45:59.664568    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:46:02.375343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:02.375310    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.713946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:05.713913    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:06.060666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:06.060577    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:07.711673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:07.711633    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:09.288569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.288532    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:09.475628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.475578    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:09.825851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.825820    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:46:09.841694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.841468    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:09.889464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.889419    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:46:09.934198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.934164    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9e18b5-dockercfg-snknj\""
Oct 30 22:46:09.934372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.934211    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9e18b5\""
Oct 30 22:46:09.941910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.941884    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hmj\" (UniqueName: \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:09.941989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:09.941917    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.042984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.042953    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hmj\" (UniqueName: \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.042984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.042987    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.043679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.043657    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.099351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.099286    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hmj\" (UniqueName: \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\") pod \"payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") " pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.163952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.163918    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.539906 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.539879    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:46:10.540914 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:46:10.540887    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f136ea1_1a30_450d_92d7_68d79ff4d2bf.slice/crio-fecc7e29e792aa1a6484bb81d44e16ddff9f89695a877136d140aadccb47feb0 WatchSource:0}: Error finding container fecc7e29e792aa1a6484bb81d44e16ddff9f89695a877136d140aadccb47feb0: Status 404 returned error can't find the container with id fecc7e29e792aa1a6484bb81d44e16ddff9f89695a877136d140aadccb47feb0
Oct 30 22:46:10.783786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.783735    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" event={"ID":"3f136ea1-1a30-450d-92d7-68d79ff4d2bf","Type":"ContainerStarted","Data":"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"}
Oct 30 22:46:10.783964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.783804    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" event={"ID":"3f136ea1-1a30-450d-92d7-68d79ff4d2bf","Type":"ContainerStarted","Data":"fecc7e29e792aa1a6484bb81d44e16ddff9f89695a877136d140aadccb47feb0"}
Oct 30 22:46:10.783964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.783837    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:10.899621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:10.899574    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" podStartSLOduration=1.8995596890000002 podStartE2EDuration="1.899559689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:10.897381982 +0000 UTC m=+3823.986119796" watchObservedRunningTime="2026-10-30 22:46:10.899559689 +0000 UTC m=+3823.988297511"
Oct 30 22:46:13.025519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:13.025485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.535290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:17.535256    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:18.325675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:18.325633    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:20.523311 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:46:20.523271    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:46:21.793726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:21.793695    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:46:23.455621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:23.455581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:24.466830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:24.466787    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:25.714636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:25.714594    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.321904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.321868    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.742665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.741939    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.911391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.911360    4410 generic.go:356] "Generic (PLEG): container finished" podID="9638907f-d147-4133-8c65-45cf5c88feca" containerID="c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95" exitCode=0
Oct 30 22:46:26.911549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.911406    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerDied","Data":"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"}
Oct 30 22:46:26.911854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.911839    4410 scope.go:122] "RemoveContainer" containerID="c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"
Oct 30 22:46:26.913453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.913421    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:26.913553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.913494    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:26.913553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:26.913526    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:27.920103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:27.920057    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerStarted","Data":"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"}
Oct 30 22:46:28.122631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:28.122594    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:46:28.122820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:28.122770    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 22:46:28.122876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:28.122841    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 22:46:28.591427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:28.591389    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:29.123111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:29.123076    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 22:46:29.123519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:29.123126    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 22:46:29.611860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:29.611827    4410 scope.go:122] "RemoveContainer" containerID="7863e59587a75570a7d4f4d90533153a472cefe27a746a46c776e798bdb03663"
Oct 30 22:46:29.622463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:29.622444    4410 scope.go:122] "RemoveContainer" containerID="5dcd26efbcfb1bfb6281df8f04adb7259151c82ce6d2cc5245cacc2f5269e3aa"
Oct 30 22:46:30.122855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:30.122821    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 22:46:30.123034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:30.122873    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 22:46:30.577908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:30.577872    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:30.771310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:30.771268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:30.924916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:30.924883    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:31.122448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:31.122417    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 22:46:31.122610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:31.122466    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 22:46:32.123271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:32.123236    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 22:46:32.123651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:32.123285    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 22:46:33.127745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:33.127719    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:46:33.128214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:33.128194    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:46:33.128756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:33.128738    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:46:33.788096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:33.788062    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:33.956385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:33.956348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:34.126799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.126692    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:34.243418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.243387    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:46:34.248769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.248749    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.304649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.304625    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b1b1eb\""
Oct 30 22:46:34.304814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.304759    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b1b1eb-dockercfg-5c58h\""
Oct 30 22:46:34.308175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.308149    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:46:34.350825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.350765    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.351014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.350844    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwm7\" (UniqueName: \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.400198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.400164    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:46:34.404487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.404465    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.452269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.452238    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbxr\" (UniqueName: \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.452402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.452290    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.452494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.452406    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.452494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.452453    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwm7\" (UniqueName: \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.453063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.453047    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.463055 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.463016    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:46:34.509110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.509083    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwm7\" (UniqueName: \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\") pod \"payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.553372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.553345    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbxr\" (UniqueName: \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.553515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.553393    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.554137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.554112    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.562204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.562186    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:34.610483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.610453    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbxr\" (UniqueName: \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\") pod \"payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.718293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.718217    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:34.942672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.942636    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:46:34.944263 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:46:34.944232    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a21e6b_00df_4c92_b1c7_0fc725fefcbc.slice/crio-0b26caa3f3d4ff583b917cca71f106c7dc3b981259cb65d48416f7f7423a8b6f WatchSource:0}: Error finding container 0b26caa3f3d4ff583b917cca71f106c7dc3b981259cb65d48416f7f7423a8b6f: Status 404 returned error can't find the container with id 0b26caa3f3d4ff583b917cca71f106c7dc3b981259cb65d48416f7f7423a8b6f
Oct 30 22:46:34.968338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:34.968283    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" event={"ID":"56a21e6b-00df-4c92-b1c7-0fc725fefcbc","Type":"ContainerStarted","Data":"0b26caa3f3d4ff583b917cca71f106c7dc3b981259cb65d48416f7f7423a8b6f"}
Oct 30 22:46:35.098570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.098542    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:46:35.100650 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:46:35.100625    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a41aea1_38f5_4645_8f3c_5e01bbdabac3.slice/crio-dac81e9d3fb0ba9465979687032ac80c4ee8f47ac19ace3967625b1fe397f958 WatchSource:0}: Error finding container dac81e9d3fb0ba9465979687032ac80c4ee8f47ac19ace3967625b1fe397f958: Status 404 returned error can't find the container with id dac81e9d3fb0ba9465979687032ac80c4ee8f47ac19ace3967625b1fe397f958
Oct 30 22:46:35.575829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.575758    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:35.976187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.976146    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" event={"ID":"56a21e6b-00df-4c92-b1c7-0fc725fefcbc","Type":"ContainerStarted","Data":"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"}
Oct 30 22:46:35.976393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.976208    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:35.977785 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.977746    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" event={"ID":"1a41aea1-38f5-4645-8f3c-5e01bbdabac3","Type":"ContainerStarted","Data":"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"}
Oct 30 22:46:35.977922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.977807    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" event={"ID":"1a41aea1-38f5-4645-8f3c-5e01bbdabac3","Type":"ContainerStarted","Data":"dac81e9d3fb0ba9465979687032ac80c4ee8f47ac19ace3967625b1fe397f958"}
Oct 30 22:46:35.977922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:35.977830    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:36.097975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:36.097928    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" podStartSLOduration=2.09791453 podStartE2EDuration="2.09791453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:36.095277184 +0000 UTC m=+3849.184014988" watchObservedRunningTime="2026-10-30 22:46:36.09791453 +0000 UTC m=+3849.186652333"
Oct 30 22:46:36.220380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:36.220307    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" podStartSLOduration=2.220286199 podStartE2EDuration="2.220286199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:36.216334553 +0000 UTC m=+3849.305072368" watchObservedRunningTime="2026-10-30 22:46:36.220286199 +0000 UTC m=+3849.309024035"
Oct 30 22:46:39.087523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:39.087481    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:44.000556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:44.000510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:44.150470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:44.150437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:44.319437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:44.319345    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:46.986203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:46.986176    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:46:46.986620 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:46.986575    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:46:49.049416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:49.049368    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:50.521147 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:46:50.521116    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:46:53.872472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:53.872432    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:58.662042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:46:58.662002    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:01.049338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:01.049301    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"]
Oct 30 22:47:01.098021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:01.097990    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035aa228-r6lwc"]
Oct 30 22:47:03.019792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:03.019748    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e405797c-afab-4a20-8a38-a7642141ce6e" path="/var/lib/kubelet/pods/e405797c-afab-4a20-8a38-a7642141ce6e/volumes"
Oct 30 22:47:03.439286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:03.439252    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:08.207308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:08.207268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:12.048688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:12.048657    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"]
Oct 30 22:47:12.097843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:12.097812    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b56146-k4jqz"]
Oct 30 22:47:12.975035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:12.974991    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:13.021732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:13.021694    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b129ce8-5517-4d01-b125-cf155d1bdd1d" path="/var/lib/kubelet/pods/4b129ce8-5517-4d01-b125-cf155d1bdd1d/volumes"
Oct 30 22:47:17.711531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:17.711494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:19.065754 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:19.065721    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.523294 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:47:20.523137    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:47:22.727477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:22.727446    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:23.052922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.052846    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"]
Oct 30 22:47:23.057671 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.057644    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.113896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.113866    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"]
Oct 30 22:47:23.159077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.159050    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:23.159226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.159100    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:23.159420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.159404    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:47:23.159494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.159450    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:23.170207 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.170182    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.170323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.170218    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94snv\" (UniqueName: \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.170425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.170410    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.271697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.271667    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.271857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.271787    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.271857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.271812    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94snv\" (UniqueName: \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.272337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.272316    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.274063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.274042    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.329830 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.329749    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94snv\" (UniqueName: \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\") pod \"maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.372113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.372091    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:23.743644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:23.743618    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"]
Oct 30 22:47:23.745282 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:47:23.745252    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e8f68_1c82_4ced_8fd7_fa04f614b81b.slice/crio-0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e WatchSource:0}: Error finding container 0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e: Status 404 returned error can't find the container with id 0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e
Oct 30 22:47:24.267440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:24.267405    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k" event={"ID":"994e8f68-1c82-4ced-8fd7-fa04f614b81b","Type":"ContainerStarted","Data":"ac7dd4dcbd9e8fd6d58bcf8ae1dc6d604ba09b3d835676332f759be7aa9eaefa"}
Oct 30 22:47:24.267589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:24.267458    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k" event={"ID":"994e8f68-1c82-4ced-8fd7-fa04f614b81b","Type":"ContainerStarted","Data":"0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e"}
Oct 30 22:47:24.469467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:24.469430    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:25.275429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:25.275387    4410 generic.go:356] "Generic (PLEG): container finished" podID="994e8f68-1c82-4ced-8fd7-fa04f614b81b" containerID="ac7dd4dcbd9e8fd6d58bcf8ae1dc6d604ba09b3d835676332f759be7aa9eaefa" exitCode=0
Oct 30 22:47:25.275850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:25.275471    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k" event={"ID":"994e8f68-1c82-4ced-8fd7-fa04f614b81b","Type":"ContainerDied","Data":"ac7dd4dcbd9e8fd6d58bcf8ae1dc6d604ba09b3d835676332f759be7aa9eaefa"}
Oct 30 22:47:26.426110 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.426086    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:26.608678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.608607    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\") pod \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") "
Oct 30 22:47:26.608678 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.608645    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\" (UniqueName: \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\") pod \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") "
Oct 30 22:47:26.608934 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.608857    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\") pod \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\" (UID: \"994e8f68-1c82-4ced-8fd7-fa04f614b81b\") "
Oct 30 22:47:26.609214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.609190    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca" pod "994e8f68-1c82-4ced-8fd7-fa04f614b81b" (UID: "994e8f68-1c82-4ced-8fd7-fa04f614b81b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:26.609351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.609335    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:26.610654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.610631    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca" pod "994e8f68-1c82-4ced-8fd7-fa04f614b81b" (UID: "994e8f68-1c82-4ced-8fd7-fa04f614b81b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:26.610755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.610697    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv" pod "994e8f68-1c82-4ced-8fd7-fa04f614b81b" (UID: "994e8f68-1c82-4ced-8fd7-fa04f614b81b"). InnerVolumeSpecName "kube-api-access-94snv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:26.710335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.710309    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e8f68-1c82-4ced-8fd7-fa04f614b81b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:26.710442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:26.710336    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94snv\" (UniqueName: \"kubernetes.io/projected/994e8f68-1c82-4ced-8fd7-fa04f614b81b-kube-api-access-94snv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:27.288434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:27.288357    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"
Oct 30 22:47:27.288434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:27.288389    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k" event={"ID":"994e8f68-1c82-4ced-8fd7-fa04f614b81b","Type":"ContainerDied","Data":"0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e"}
Oct 30 22:47:27.288607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:27.288437    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfce9fe753619e8e04f83c4e6439356f8c9009e0a741e1d98c1394e57b65e1e"
Oct 30 22:47:27.928545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:27.928511    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:33.071603 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:33.071571    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.665988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:34.665951    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:47:34.936332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:34.936250    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.092692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:35.092659    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.899157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:35.899128    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:37.838017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:37.837975    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:39.666368 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.666317    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" podUID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" containerName="payload-pre-processing" containerID="cri-o://c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9" gracePeriod=25
Oct 30 22:47:39.917736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.917684    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:47:39.955175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.955149    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\") pod \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") "
Oct 30 22:47:39.955309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.955217    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\" (UniqueName: \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\") pod \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\" (UID: \"36d9bd92-e53b-493f-9293-408d0b9dd4f4\") "
Oct 30 22:47:39.955484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.955462    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume" pod "36d9bd92-e53b-493f-9293-408d0b9dd4f4" (UID: "36d9bd92-e53b-493f-9293-408d0b9dd4f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:39.955622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.955607    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d9bd92-e53b-493f-9293-408d0b9dd4f4-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:39.957278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:39.957255    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng" pod "36d9bd92-e53b-493f-9293-408d0b9dd4f4" (UID: "36d9bd92-e53b-493f-9293-408d0b9dd4f4"). InnerVolumeSpecName "kube-api-access-gp5ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:40.056719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.056683    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp5ng\" (UniqueName: \"kubernetes.io/projected/36d9bd92-e53b-493f-9293-408d0b9dd4f4-kube-api-access-gp5ng\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.371726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.371649    4410 generic.go:356] "Generic (PLEG): container finished" podID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" containerID="c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9" exitCode=0
Oct 30 22:47:40.371726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.371698    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"
Oct 30 22:47:40.371973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.371728    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" event={"ID":"36d9bd92-e53b-493f-9293-408d0b9dd4f4","Type":"ContainerDied","Data":"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"}
Oct 30 22:47:40.371973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.371799    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9" event={"ID":"36d9bd92-e53b-493f-9293-408d0b9dd4f4","Type":"ContainerDied","Data":"3aafe30f5b8e8637a898611d7a98533b572570ed43eb11f1a25947232ae9a1a9"}
Oct 30 22:47:40.371973 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.371821    4410 scope.go:122] "RemoveContainer" containerID="c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"
Oct 30 22:47:40.381544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.381525    4410 scope.go:122] "RemoveContainer" containerID="c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"
Oct 30 22:47:40.381824 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:47:40.381802    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9\": container with ID starting with c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9 not found: ID does not exist" containerID="c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"
Oct 30 22:47:40.381884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.381832    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9"} err="failed to get container status \"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9\": rpc error: code = NotFound desc = could not find container \"c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9\": container with ID starting with c45991d4c6a3cb8b08d5c04dd1cdc2dee4cf0cdbe8b1019bfd3c8a456cd7e7b9 not found: ID does not exist"
Oct 30 22:47:40.534842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.534802    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:47:40.539979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:40.539952    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55edb1-768b8c66c5-8q6g9"]
Oct 30 22:47:41.019871 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:41.019835    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" path="/var/lib/kubelet/pods/36d9bd92-e53b-493f-9293-408d0b9dd4f4/volumes"
Oct 30 22:47:42.767273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:42.767234    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:46.063892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:46.063857    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.051688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:47.051658    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"]
Oct 30 22:47:47.100795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:47.100757    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679796a9-77skn"]
Oct 30 22:47:47.539071 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:47.539031    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.757032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:47.756998    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.920430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:47.920399    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:49.019349 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:49.019314    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df2245e-8128-4b62-ad43-f6b802c36442" path="/var/lib/kubelet/pods/7df2245e-8128-4b62-ad43-f6b802c36442/volumes"
Oct 30 22:47:50.521560 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:47:50.521521    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:47:52.585571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:52.585536    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.483128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:57.483088    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:58.048634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:58.048605    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"]
Oct 30 22:47:58.081777 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:58.081752    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541524d6-66mmw"]
Oct 30 22:47:59.020921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:47:59.020890    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78022a8e-94ca-42d7-bbca-2264b8d6ec31" path="/var/lib/kubelet/pods/78022a8e-94ca-42d7-bbca-2264b8d6ec31/volumes"
Oct 30 22:48:02.364024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:02.363989    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:07.131239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:07.131203    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:07.286761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:07.286718    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.911418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:11.911376    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.651087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:16.651053    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.800572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:16.800533    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.946713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:16.946628    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:17.126618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.126586    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"]
Oct 30 22:48:17.127184 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127164    4410 cpu_manager.go:414] "containerMap: removing container" podUID="994e8f68-1c82-4ced-8fd7-fa04f614b81b" containerName="revoke-keys"
Oct 30 22:48:17.127303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127186    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994e8f68-1c82-4ced-8fd7-fa04f614b81b" containerName="revoke-keys"
Oct 30 22:48:17.127303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127210    4410 cpu_manager.go:414] "containerMap: removing container" podUID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" containerName="payload-pre-processing"
Oct 30 22:48:17.127303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127218    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" containerName="payload-pre-processing"
Oct 30 22:48:17.127465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127391    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="994e8f68-1c82-4ced-8fd7-fa04f614b81b" containerName="revoke-keys"
Oct 30 22:48:17.127465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.127406    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d9bd92-e53b-493f-9293-408d0b9dd4f4" containerName="payload-pre-processing"
Oct 30 22:48:17.131297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.131271    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.194360 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.194331    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:17.233907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.233806    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:17.233907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.233805    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:17.234175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.234043    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:48:17.236933 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.236909    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"]
Oct 30 22:48:17.250569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.250523    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5xv\" (UniqueName: \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.250761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.250740    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.250865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.250781    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.351943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.351913    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5xv\" (UniqueName: \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.352087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.352026    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.352087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.352044    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.352680 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.352648    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.354407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.354387    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.409128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.409108    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5xv\" (UniqueName: \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\") pod \"maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.445970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.445946    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:17.828313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:17.828268    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"]
Oct 30 22:48:17.829803 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:48:17.829762    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb62ad1_a22f_4010_ab81_c8b6d935d781.slice/crio-f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26 WatchSource:0}: Error finding container f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26: Status 404 returned error can't find the container with id f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26
Oct 30 22:48:18.609204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:18.609112    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v" event={"ID":"1cb62ad1-a22f-4010-ab81-c8b6d935d781","Type":"ContainerStarted","Data":"bf95ee16a9a51fdd7438cadf89c2f83cf09dfe742f14bdbfad2fe62a7b61f44b"}
Oct 30 22:48:18.609204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:18.609174    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v" event={"ID":"1cb62ad1-a22f-4010-ab81-c8b6d935d781","Type":"ContainerStarted","Data":"f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26"}
Oct 30 22:48:18.723342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:18.723280    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v" podStartSLOduration=1.7232602460000002 podStartE2EDuration="1.723260246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:18.721006042 +0000 UTC m=+3951.809743861" watchObservedRunningTime="2026-10-30 22:48:18.723260246 +0000 UTC m=+3951.811998052"
Oct 30 22:48:19.618706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:19.618673    4410 generic.go:356] "Generic (PLEG): container finished" podID="1cb62ad1-a22f-4010-ab81-c8b6d935d781" containerID="bf95ee16a9a51fdd7438cadf89c2f83cf09dfe742f14bdbfad2fe62a7b61f44b" exitCode=0
Oct 30 22:48:19.619210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:19.618759    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v" event={"ID":"1cb62ad1-a22f-4010-ab81-c8b6d935d781","Type":"ContainerDied","Data":"bf95ee16a9a51fdd7438cadf89c2f83cf09dfe742f14bdbfad2fe62a7b61f44b"}
Oct 30 22:48:20.522841 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:48:20.522799    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:48:20.772507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.772485    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:20.888715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.888694    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\" (UniqueName: \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\") pod \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") "
Oct 30 22:48:20.888872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.888858    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\") pod \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") "
Oct 30 22:48:20.888916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.888902    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\") pod \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\" (UID: \"1cb62ad1-a22f-4010-ab81-c8b6d935d781\") "
Oct 30 22:48:20.889271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.889246    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca" pod "1cb62ad1-a22f-4010-ab81-c8b6d935d781" (UID: "1cb62ad1-a22f-4010-ab81-c8b6d935d781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:20.889367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.889345    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:20.890725 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.890706    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv" pod "1cb62ad1-a22f-4010-ab81-c8b6d935d781" (UID: "1cb62ad1-a22f-4010-ab81-c8b6d935d781"). InnerVolumeSpecName "kube-api-access-wn5xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:20.891146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.891127    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca" pod "1cb62ad1-a22f-4010-ab81-c8b6d935d781" (UID: "1cb62ad1-a22f-4010-ab81-c8b6d935d781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:20.990084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.990050    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb62ad1-a22f-4010-ab81-c8b6d935d781-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:20.990084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:20.990079    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5xv\" (UniqueName: \"kubernetes.io/projected/1cb62ad1-a22f-4010-ab81-c8b6d935d781-kube-api-access-wn5xv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:21.639293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:21.639260    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"
Oct 30 22:48:21.639293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:21.639266    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v" event={"ID":"1cb62ad1-a22f-4010-ab81-c8b6d935d781","Type":"ContainerDied","Data":"f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26"}
Oct 30 22:48:21.639499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:21.639354    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f912ec2d585fb88cb05a312f350a0be9d070399d9cf8838b9e5373336d2cbc26"
Oct 30 22:48:22.132461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:22.132429    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:27.358718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:27.358677    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:29.035670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:29.035635    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:48:29.416444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:29.416409    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:29.620508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:29.620465    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:29.656953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:29.656927    4410 scope.go:122] "RemoveContainer" containerID="3c39e94d8efcfa9e4a9de4cd0071eba502d6f3fcb8f8f237cc8480f8a3fb9ccc"
Oct 30 22:48:29.666651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:29.666633    4410 scope.go:122] "RemoveContainer" containerID="2384adaccbffb4ae15821de989197cf0e872e79fdce48117f0fc6bbefa42c1f7"
Oct 30 22:48:32.281043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:32.281011    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:34.035903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.035870    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" podUID="d232b960-03da-47b7-a60c-89d02fa9117c" containerName="payload-processing" containerID="cri-o://d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90" gracePeriod=25
Oct 30 22:48:34.299256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.299203    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:48:34.333842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.333815    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\") pod \"d232b960-03da-47b7-a60c-89d02fa9117c\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") "
Oct 30 22:48:34.333971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.333949    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\" (UniqueName: \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\") pod \"d232b960-03da-47b7-a60c-89d02fa9117c\" (UID: \"d232b960-03da-47b7-a60c-89d02fa9117c\") "
Oct 30 22:48:34.334267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.334239    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume" pod "d232b960-03da-47b7-a60c-89d02fa9117c" (UID: "d232b960-03da-47b7-a60c-89d02fa9117c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:34.334380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.334355    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b960-03da-47b7-a60c-89d02fa9117c-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.335916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.335892    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg" pod "d232b960-03da-47b7-a60c-89d02fa9117c" (UID: "d232b960-03da-47b7-a60c-89d02fa9117c"). InnerVolumeSpecName "kube-api-access-phvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:34.435930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.435895    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvmg\" (UniqueName: \"kubernetes.io/projected/d232b960-03da-47b7-a60c-89d02fa9117c-kube-api-access-phvmg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.747052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.746876    4410 generic.go:356] "Generic (PLEG): container finished" podID="d232b960-03da-47b7-a60c-89d02fa9117c" containerID="d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90" exitCode=0
Oct 30 22:48:34.747052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.746929    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"
Oct 30 22:48:34.747052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.746978    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" event={"ID":"d232b960-03da-47b7-a60c-89d02fa9117c","Type":"ContainerDied","Data":"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"}
Oct 30 22:48:34.747052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.747023    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8" event={"ID":"d232b960-03da-47b7-a60c-89d02fa9117c","Type":"ContainerDied","Data":"a6e4e8a62fe6a04a504348c463bc5499feffff5c38b185710e4ea4bce8783828"}
Oct 30 22:48:34.747052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.747050    4410 scope.go:122] "RemoveContainer" containerID="d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"
Oct 30 22:48:34.756965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.756942    4410 scope.go:122] "RemoveContainer" containerID="d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"
Oct 30 22:48:34.757237 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:48:34.757214    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90\": container with ID starting with d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90 not found: ID does not exist" containerID="d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"
Oct 30 22:48:34.757294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.757245    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90"} err="failed to get container status \"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90\": rpc error: code = NotFound desc = could not find container \"d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90\": container with ID starting with d39ebcdcffe565ae7a6ec387ba70a8a6fdcdaeddcad82683392823c238b57a90 not found: ID does not exist"
Oct 30 22:48:34.909655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.909626    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:48:34.959079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:34.959055    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a6900c-7686bc5445-fbfd8"]
Oct 30 22:48:35.019337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:35.019265    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d232b960-03da-47b7-a60c-89d02fa9117c" path="/var/lib/kubelet/pods/d232b960-03da-47b7-a60c-89d02fa9117c/volumes"
Oct 30 22:48:36.229497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:36.229457    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:36.382752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:36.382716    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:37.158866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:37.158832    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:42.356470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:42.356432    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:46.682338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682302    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"]
Oct 30 22:48:46.682854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682838    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d232b960-03da-47b7-a60c-89d02fa9117c" containerName="payload-processing"
Oct 30 22:48:46.682911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682854    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d232b960-03da-47b7-a60c-89d02fa9117c" containerName="payload-processing"
Oct 30 22:48:46.682911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682874    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1cb62ad1-a22f-4010-ab81-c8b6d935d781" containerName="revoke-keys"
Oct 30 22:48:46.682911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682880    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb62ad1-a22f-4010-ab81-c8b6d935d781" containerName="revoke-keys"
Oct 30 22:48:46.683011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682960    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d232b960-03da-47b7-a60c-89d02fa9117c" containerName="payload-processing"
Oct 30 22:48:46.683011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.682972    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb62ad1-a22f-4010-ab81-c8b6d935d781" containerName="revoke-keys"
Oct 30 22:48:46.686386 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.686368    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.739176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.739150    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:46.741660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.741640    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:48:46.742473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.742446    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:46.762981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.762950    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.763135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.763009    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.763135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.763114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pj4\" (UniqueName: \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.788877 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.788848    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:46.791660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.791639    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"]
Oct 30 22:48:46.864586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.864543    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.864586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.864600    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.864842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.864626    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pj4\" (UniqueName: \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.865270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.865248    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.867156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.867129    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:46.922104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:46.922076    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pj4\" (UniqueName: \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\") pod \"maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:47.002043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.001967    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:47.377597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.377566    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"]
Oct 30 22:48:47.379077 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:48:47.379051    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba79a582_5b82_46bb_b993_2d847b808631.slice/crio-9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c WatchSource:0}: Error finding container 9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c: Status 404 returned error can't find the container with id 9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c
Oct 30 22:48:47.673411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.673386    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:47.830179 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.830146    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv" event={"ID":"ba79a582-5b82-46bb-b993-2d847b808631","Type":"ContainerStarted","Data":"5c87bf66816a9f45f92ad73de05ec72139703f729508be83cf8f2b4974ce9524"}
Oct 30 22:48:47.830616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.830198    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv" event={"ID":"ba79a582-5b82-46bb-b993-2d847b808631","Type":"ContainerStarted","Data":"9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c"}
Oct 30 22:48:47.944756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:47.944652    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv" podStartSLOduration=1.9446379139999999 podStartE2EDuration="1.944637914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:47.943383423 +0000 UTC m=+3981.032121225" watchObservedRunningTime="2026-10-30 22:48:47.944637914 +0000 UTC m=+3981.033375722"
Oct 30 22:48:48.837270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:48.837235    4410 generic.go:356] "Generic (PLEG): container finished" podID="ba79a582-5b82-46bb-b993-2d847b808631" containerID="5c87bf66816a9f45f92ad73de05ec72139703f729508be83cf8f2b4974ce9524" exitCode=0
Oct 30 22:48:48.837804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:48.837284    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv" event={"ID":"ba79a582-5b82-46bb-b993-2d847b808631","Type":"ContainerDied","Data":"5c87bf66816a9f45f92ad73de05ec72139703f729508be83cf8f2b4974ce9524"}
Oct 30 22:48:49.990469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:49.990441    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:50.100139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.100073    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\" (UniqueName: \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\") pod \"ba79a582-5b82-46bb-b993-2d847b808631\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") "
Oct 30 22:48:50.100139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.100108    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\") pod \"ba79a582-5b82-46bb-b993-2d847b808631\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") "
Oct 30 22:48:50.100331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.100140    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\") pod \"ba79a582-5b82-46bb-b993-2d847b808631\" (UID: \"ba79a582-5b82-46bb-b993-2d847b808631\") "
Oct 30 22:48:50.100439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.100415    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca" pod "ba79a582-5b82-46bb-b993-2d847b808631" (UID: "ba79a582-5b82-46bb-b993-2d847b808631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:50.100795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.100748    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.102239 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.102216    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca" pod "ba79a582-5b82-46bb-b993-2d847b808631" (UID: "ba79a582-5b82-46bb-b993-2d847b808631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:50.102335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.102292    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4" pod "ba79a582-5b82-46bb-b993-2d847b808631" (UID: "ba79a582-5b82-46bb-b993-2d847b808631"). InnerVolumeSpecName "kube-api-access-j9pj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:50.202051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.202020    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pj4\" (UniqueName: \"kubernetes.io/projected/ba79a582-5b82-46bb-b993-2d847b808631-kube-api-access-j9pj4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.202051 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.202048    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba79a582-5b82-46bb-b993-2d847b808631-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.523088 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:48:50.523056    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:48:50.852128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.852055    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"
Oct 30 22:48:50.852128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.852064    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv" event={"ID":"ba79a582-5b82-46bb-b993-2d847b808631","Type":"ContainerDied","Data":"9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c"}
Oct 30 22:48:50.852128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:50.852113    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b94f98391fe581bfdbfdb582196aad06a997874240d6c1017f476f251a7f66c"
Oct 30 22:48:52.852798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:52.852740    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:57.771602 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:57.771565    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.324186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:58.324151    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:48:58.603740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:58.603656    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.799304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:48:58.799266    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:02.605481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:02.605449    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:03.325117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.325073    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" podUID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" containerName="payload-pre-processing" containerID="cri-o://f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa" gracePeriod=25
Oct 30 22:49:03.580695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.580635    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:49:03.663340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.663315    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\" (UniqueName: \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\") pod \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") "
Oct 30 22:49:03.663751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.663426    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\") pod \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\" (UID: \"6c64a939-a39d-4b45-8924-b15c6ac5eb40\") "
Oct 30 22:49:03.663879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.663854    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume" pod "6c64a939-a39d-4b45-8924-b15c6ac5eb40" (UID: "6c64a939-a39d-4b45-8924-b15c6ac5eb40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:03.665379 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.665356    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm" pod "6c64a939-a39d-4b45-8924-b15c6ac5eb40" (UID: "6c64a939-a39d-4b45-8924-b15c6ac5eb40"). InnerVolumeSpecName "kube-api-access-9mxcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:03.765056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.765029    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxcm\" (UniqueName: \"kubernetes.io/projected/6c64a939-a39d-4b45-8924-b15c6ac5eb40-kube-api-access-9mxcm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.765056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.765055    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c64a939-a39d-4b45-8924-b15c6ac5eb40-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.927351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.927315    4410 generic.go:356] "Generic (PLEG): container finished" podID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" containerID="f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa" exitCode=0
Oct 30 22:49:03.927514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.927360    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"
Oct 30 22:49:03.927514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.927399    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" event={"ID":"6c64a939-a39d-4b45-8924-b15c6ac5eb40","Type":"ContainerDied","Data":"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"}
Oct 30 22:49:03.927514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.927454    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q" event={"ID":"6c64a939-a39d-4b45-8924-b15c6ac5eb40","Type":"ContainerDied","Data":"bb5a49ee13966f5cba33310aed9ad3114a47576346e3c4f3cf2145f35dd485b6"}
Oct 30 22:49:03.927514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.927482    4410 scope.go:122] "RemoveContainer" containerID="f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"
Oct 30 22:49:03.937440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.937423    4410 scope.go:122] "RemoveContainer" containerID="f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"
Oct 30 22:49:03.937729 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:03.937707    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa\": container with ID starting with f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa not found: ID does not exist" containerID="f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"
Oct 30 22:49:03.937850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:03.937738    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa"} err="failed to get container status \"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa\": rpc error: code = NotFound desc = could not find container \"f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa\": container with ID starting with f4007291023620c44c903f7f9ab3c1ed87f48bc98b01abf61c3dedc7ef6202fa not found: ID does not exist"
Oct 30 22:49:04.089942 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:04.089903    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:49:04.139166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:04.139134    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20ccef-97857d599-xlm6q"]
Oct 30 22:49:05.020260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:05.020228    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" path="/var/lib/kubelet/pods/6c64a939-a39d-4b45-8924-b15c6ac5eb40/volumes"
Oct 30 22:49:07.441133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:07.441091    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:09.746789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.746748    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"]
Oct 30 22:49:09.747284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747267    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" containerName="payload-pre-processing"
Oct 30 22:49:09.747328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747284    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" containerName="payload-pre-processing"
Oct 30 22:49:09.747328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747300    4410 cpu_manager.go:414] "containerMap: removing container" podUID="ba79a582-5b82-46bb-b993-2d847b808631" containerName="revoke-keys"
Oct 30 22:49:09.747328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747305    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba79a582-5b82-46bb-b993-2d847b808631" containerName="revoke-keys"
Oct 30 22:49:09.747433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747386    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c64a939-a39d-4b45-8924-b15c6ac5eb40" containerName="payload-pre-processing"
Oct 30 22:49:09.747433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.747396    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba79a582-5b82-46bb-b993-2d847b808631" containerName="revoke-keys"
Oct 30 22:49:09.750440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.750424    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.807859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.807832    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"]
Oct 30 22:49:09.827312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.827287    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.827442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.827352    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.827614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.827584    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrl7\" (UniqueName: \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.853446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.853385    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:09.853446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.853384    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:09.853608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.853443    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:09.853608 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.853518    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:49:09.929121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.929093    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrl7\" (UniqueName: \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.929291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.929141    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.929291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.929206    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.929825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.929804    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.931425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.931402    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:09.985426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:09.985401    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrl7\" (UniqueName: \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\") pod \"maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:10.063941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:10.063880    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:10.434797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:10.434754    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"]
Oct 30 22:49:10.436555 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:49:10.436526    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe61f394_afca_4a08_a9c8_6695648130c2.slice/crio-8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b WatchSource:0}: Error finding container 8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b: Status 404 returned error can't find the container with id 8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b
Oct 30 22:49:10.971827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:10.971792    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k" event={"ID":"fe61f394-afca-4a08-a9c8-6695648130c2","Type":"ContainerStarted","Data":"3d8a0dfb3119f4f7d9d5d636a54879876a78c5fbd7c79676c17442d0a1561fd8"}
Oct 30 22:49:10.972269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:10.971845    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k" event={"ID":"fe61f394-afca-4a08-a9c8-6695648130c2","Type":"ContainerStarted","Data":"8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b"}
Oct 30 22:49:11.098160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:11.098067    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k" podStartSLOduration=2.09805339 podStartE2EDuration="2.09805339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:11.09685454 +0000 UTC m=+4004.185592345" watchObservedRunningTime="2026-10-30 22:49:11.09805339 +0000 UTC m=+4004.186791257"
Oct 30 22:49:11.979373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:11.979339    4410 generic.go:356] "Generic (PLEG): container finished" podID="fe61f394-afca-4a08-a9c8-6695648130c2" containerID="3d8a0dfb3119f4f7d9d5d636a54879876a78c5fbd7c79676c17442d0a1561fd8" exitCode=0
Oct 30 22:49:11.979791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:11.979425    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k" event={"ID":"fe61f394-afca-4a08-a9c8-6695648130c2","Type":"ContainerDied","Data":"3d8a0dfb3119f4f7d9d5d636a54879876a78c5fbd7c79676c17442d0a1561fd8"}
Oct 30 22:49:12.575479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:12.575437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:13.135204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.135181    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:13.263614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.263545    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\") pod \"fe61f394-afca-4a08-a9c8-6695648130c2\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") "
Oct 30 22:49:13.263848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.263620    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\" (UniqueName: \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\") pod \"fe61f394-afca-4a08-a9c8-6695648130c2\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") "
Oct 30 22:49:13.263848 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.263681    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\") pod \"fe61f394-afca-4a08-a9c8-6695648130c2\" (UID: \"fe61f394-afca-4a08-a9c8-6695648130c2\") "
Oct 30 22:49:13.264064 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.264031    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca" pod "fe61f394-afca-4a08-a9c8-6695648130c2" (UID: "fe61f394-afca-4a08-a9c8-6695648130c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:13.264305 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.264279    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:13.266914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.266885    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca" pod "fe61f394-afca-4a08-a9c8-6695648130c2" (UID: "fe61f394-afca-4a08-a9c8-6695648130c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:13.267024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.266938    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7" pod "fe61f394-afca-4a08-a9c8-6695648130c2" (UID: "fe61f394-afca-4a08-a9c8-6695648130c2"). InnerVolumeSpecName "kube-api-access-wtrl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.365326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.365297    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrl7\" (UniqueName: \"kubernetes.io/projected/fe61f394-afca-4a08-a9c8-6695648130c2-kube-api-access-wtrl7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:13.365326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.365324    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe61f394-afca-4a08-a9c8-6695648130c2-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:13.992129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.992103    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"
Oct 30 22:49:13.992318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.992099    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k" event={"ID":"fe61f394-afca-4a08-a9c8-6695648130c2","Type":"ContainerDied","Data":"8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b"}
Oct 30 22:49:13.992318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:13.992240    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adde424974527557b1e3fc376611c59219647e65cd14f13c581ac8296edff8b"
Oct 30 22:49:17.859495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:17.859462    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.522620 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:20.522579    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:49:21.292464 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:21.292433    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:49:21.680949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:21.680919    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:21.841549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:21.841515    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:22.473333 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:22.473289    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.171554 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:24.171516    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.398962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:24.398926    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.546924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:24.546842    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.814290 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:24.814212    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"]
Oct 30 22:49:24.863269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:24.863238    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-55edb1-mm57k"]
Oct 30 22:49:25.020352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:25.020310    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994e8f68-1c82-4ced-8fd7-fa04f614b81b" path="/var/lib/kubelet/pods/994e8f68-1c82-4ced-8fd7-fa04f614b81b/volumes"
Oct 30 22:49:25.258502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:25.258471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:26.293276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.293238    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" podUID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" containerName="payload-processing" containerID="cri-o://f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb" gracePeriod=25
Oct 30 22:49:26.560136 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.560077    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:49:26.706462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.706423    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\" (UniqueName: \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\") pod \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") "
Oct 30 22:49:26.706655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.706622    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\") pod \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\" (UID: \"3f136ea1-1a30-450d-92d7-68d79ff4d2bf\") "
Oct 30 22:49:26.706991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.706969    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume" pod "3f136ea1-1a30-450d-92d7-68d79ff4d2bf" (UID: "3f136ea1-1a30-450d-92d7-68d79ff4d2bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:26.709080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.709059    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj" pod "3f136ea1-1a30-450d-92d7-68d79ff4d2bf" (UID: "3f136ea1-1a30-450d-92d7-68d79ff4d2bf"). InnerVolumeSpecName "kube-api-access-46hmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:26.808504 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.808470    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.808633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:26.808508    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hmj\" (UniqueName: \"kubernetes.io/projected/3f136ea1-1a30-450d-92d7-68d79ff4d2bf-kube-api-access-46hmj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:27.070322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.070290    4410 generic.go:356] "Generic (PLEG): container finished" podID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" containerID="f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb" exitCode=0
Oct 30 22:49:27.070475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.070332    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"
Oct 30 22:49:27.070475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.070372    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" event={"ID":"3f136ea1-1a30-450d-92d7-68d79ff4d2bf","Type":"ContainerDied","Data":"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"}
Oct 30 22:49:27.070475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.070425    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98" event={"ID":"3f136ea1-1a30-450d-92d7-68d79ff4d2bf","Type":"ContainerDied","Data":"fecc7e29e792aa1a6484bb81d44e16ddff9f89695a877136d140aadccb47feb0"}
Oct 30 22:49:27.070475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.070446    4410 scope.go:122] "RemoveContainer" containerID="f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"
Oct 30 22:49:27.079966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.079946    4410 scope.go:122] "RemoveContainer" containerID="f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"
Oct 30 22:49:27.080236 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:27.080216    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb\": container with ID starting with f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb not found: ID does not exist" containerID="f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"
Oct 30 22:49:27.080330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.080243    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb"} err="failed to get container status \"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb\": rpc error: code = NotFound desc = could not find container \"f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb\": container with ID starting with f36d79a76bce4f9f24b3d0844c2b55185bff1cae3edc48837b230b899fdbf9cb not found: ID does not exist"
Oct 30 22:49:27.233707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.233676    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:49:27.237821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:27.237790    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9e18b5-6875cb8497-7tp98"]
Oct 30 22:49:29.020590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:29.020554    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" path="/var/lib/kubelet/pods/3f136ea1-1a30-450d-92d7-68d79ff4d2bf/volumes"
Oct 30 22:49:29.698444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:29.698415    4410 scope.go:122] "RemoveContainer" containerID="75f86767fdc5e259e10cba09187c0ec95111b4394bfc08055695acdf62b2c7d8"
Oct 30 22:49:29.708053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:29.708027    4410 scope.go:122] "RemoveContainer" containerID="353f384671c40e58bd65831fab6fb0a636c4c43f7097dba812affcc71505ce28"
Oct 30 22:49:29.729303 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:29.729272    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.393490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:31.393448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.544174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:31.544132    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.711442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:31.711359    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:49:31.711595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:31.711563    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" containerID="cri-o://ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3" gracePeriod=30
Oct 30 22:49:31.729048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:31.729021    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:33.130468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.130436    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:49:33.130899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.130488    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:49:33.982536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.982501    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"]
Oct 30 22:49:33.983011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.982997    4410 cpu_manager.go:414] "containerMap: removing container" podUID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" containerName="payload-processing"
Oct 30 22:49:33.983068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.983012    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" containerName="payload-processing"
Oct 30 22:49:33.983068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.983039    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fe61f394-afca-4a08-a9c8-6695648130c2" containerName="revoke-keys"
Oct 30 22:49:33.983068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.983045    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe61f394-afca-4a08-a9c8-6695648130c2" containerName="revoke-keys"
Oct 30 22:49:33.983187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.983127    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe61f394-afca-4a08-a9c8-6695648130c2" containerName="revoke-keys"
Oct 30 22:49:33.983187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.983143    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f136ea1-1a30-450d-92d7-68d79ff4d2bf" containerName="payload-processing"
Oct 30 22:49:33.987910 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:33.987891    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.040850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.040608    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:34.083619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.083589    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.083766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.083643    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.083766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.083730    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thggj\" (UniqueName: \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.089700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.089678    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:49:34.089831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.089727    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:34.089831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.089748    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:34.092377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.092353    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"]
Oct 30 22:49:34.184945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.184917    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.185428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.184992    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thggj\" (UniqueName: \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.185428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.185084    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.185710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.185686    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.187545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.187437    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.241892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.241828    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thggj\" (UniqueName: \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.272869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.272827    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:34.301704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.301678    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:34.373977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.369361    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"]
Oct 30 22:49:34.376306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.376280    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.432327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.432289    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"]
Oct 30 22:49:34.492824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.492696    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.493007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.492960    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.493080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.493021    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wmh\" (UniqueName: \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.594403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.594366    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.594591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.594442    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wmh\" (UniqueName: \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.594591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.594486    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.595041 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.595022    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.596936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.596917    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.650999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.650975    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wmh\" (UniqueName: \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\") pod \"maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:34.693647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.693625    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"]
Oct 30 22:49:34.695390 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:49:34.695360    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ae1a4d_b683_4bfa_a4db_72f9e5fe424c.slice/crio-ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca WatchSource:0}: Error finding container ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca: Status 404 returned error can't find the container with id ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca
Oct 30 22:49:34.696019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:34.696001    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:35.080681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.080655    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"]
Oct 30 22:49:35.081834 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:49:35.081811    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc1f7c4_ea7d_4267_8a1a_e5ce50684244.slice/crio-258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32 WatchSource:0}: Error finding container 258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32: Status 404 returned error can't find the container with id 258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32
Oct 30 22:49:35.125393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.125361    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r" event={"ID":"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c","Type":"ContainerStarted","Data":"eb3b5c4df498e4915bd7cb411c442f9be0b8c3628054410ef06e71722513eac3"}
Oct 30 22:49:35.125526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.125422    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r" event={"ID":"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c","Type":"ContainerStarted","Data":"ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca"}
Oct 30 22:49:35.127821 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.127796    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b" event={"ID":"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244","Type":"ContainerStarted","Data":"258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32"}
Oct 30 22:49:35.241221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.241179    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r" podStartSLOduration=2.241165925 podStartE2EDuration="2.241165925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:35.2384089 +0000 UTC m=+4028.327146702" watchObservedRunningTime="2026-10-30 22:49:35.241165925 +0000 UTC m=+4028.329903728"
Oct 30 22:49:35.904582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:35.904554    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.135882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.135846    4410 generic.go:356] "Generic (PLEG): container finished" podID="d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" containerID="eb3b5c4df498e4915bd7cb411c442f9be0b8c3628054410ef06e71722513eac3" exitCode=0
Oct 30 22:49:36.136058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.135931    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r" event={"ID":"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c","Type":"ContainerDied","Data":"eb3b5c4df498e4915bd7cb411c442f9be0b8c3628054410ef06e71722513eac3"}
Oct 30 22:49:36.138208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.138186    4410 generic.go:356] "Generic (PLEG): container finished" podID="0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" containerID="9f36cfb7abd1d5f8431f9ae3d239833ae391771df840ed68aab458eb5c55ce0a" exitCode=0
Oct 30 22:49:36.138312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.138215    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b" event={"ID":"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244","Type":"ContainerDied","Data":"9f36cfb7abd1d5f8431f9ae3d239833ae391771df840ed68aab458eb5c55ce0a"}
Oct 30 22:49:36.182356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.182292    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.335841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.335804    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.399526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.399490    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"]
Oct 30 22:49:36.404874 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.404852    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e18b5-9hw2k"]
Oct 30 22:49:36.587611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.587525    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.771537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.771509    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:36.972364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:36.972341    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:49:37.019101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.019072    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe61f394-afca-4a08-a9c8-6695648130c2" path="/var/lib/kubelet/pods/fe61f394-afca-4a08-a9c8-6695648130c2/volumes"
Oct 30 22:49:37.021006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.020981    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021030    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021078    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021187 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021131    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021215    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021223    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.021351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021248    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021282    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021355    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021391    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\") pod \"9638907f-d147-4133-8c65-45cf5c88feca\" (UID: \"9638907f-d147-4133-8c65-45cf5c88feca\") "
Oct 30 22:49:37.021465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021427    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.021643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021510    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:37.021709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021641    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.021838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021812    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.021986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021970    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.022050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.021988    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638907f-d147-4133-8c65-45cf5c88feca-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.022050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.022018    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.022050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.022034    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.022050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.022049    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.023389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.023370    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "kube-api-access-hvhq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.023549 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.023530    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.023658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.023640    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:37.023764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.023678    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy" pod "9638907f-d147-4133-8c65-45cf5c88feca" (UID: "9638907f-d147-4133-8c65-45cf5c88feca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.123419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.123387    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.123419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.123413    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvhq6\" (UniqueName: \"kubernetes.io/projected/9638907f-d147-4133-8c65-45cf5c88feca-kube-api-access-hvhq6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.123419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.123425    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638907f-d147-4133-8c65-45cf5c88feca-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.123628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.123434    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638907f-d147-4133-8c65-45cf5c88feca-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.145851 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.145816    4410 generic.go:356] "Generic (PLEG): container finished" podID="9638907f-d147-4133-8c65-45cf5c88feca" containerID="ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3" exitCode=0
Oct 30 22:49:37.145992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.145910    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"
Oct 30 22:49:37.145992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.145941    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerDied","Data":"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"}
Oct 30 22:49:37.145992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.145989    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb" event={"ID":"9638907f-d147-4133-8c65-45cf5c88feca","Type":"ContainerDied","Data":"6776019031ee55a796a85275be4ce485bb507a02d6735ac7c5c0c4acc4bfc38b"}
Oct 30 22:49:37.146108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.146013    4410 scope.go:122] "RemoveContainer" containerID="ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"
Oct 30 22:49:37.156978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.156956    4410 scope.go:122] "RemoveContainer" containerID="c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"
Oct 30 22:49:37.168439 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.168415    4410 scope.go:122] "RemoveContainer" containerID="ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"
Oct 30 22:49:37.168765 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:37.168739    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3\": container with ID starting with ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3 not found: ID does not exist" containerID="ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"
Oct 30 22:49:37.168908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.168786    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3"} err="failed to get container status \"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3\": rpc error: code = NotFound desc = could not find container \"ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3\": container with ID starting with ff9299fcbeb291b33bb619105cd900cc0d7e94305cce44ec2904aee0f130b5b3 not found: ID does not exist"
Oct 30 22:49:37.168908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.168812    4410 scope.go:122] "RemoveContainer" containerID="c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"
Oct 30 22:49:37.169102 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:37.169082    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95\": container with ID starting with c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95 not found: ID does not exist" containerID="c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"
Oct 30 22:49:37.169241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.169109    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95"} err="failed to get container status \"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95\": rpc error: code = NotFound desc = could not find container \"c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95\": container with ID starting with c789b1039be13cde0aeff3d196446b78d065e2752f8096aa9d7d9975ca146d95 not found: ID does not exist"
Oct 30 22:49:37.276182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.276156    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:37.317744 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.317720    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:37.325992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.325973    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\") pod \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") "
Oct 30 22:49:37.326105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.326091    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\" (UniqueName: \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\") pod \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") "
Oct 30 22:49:37.326238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.326224    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\") pod \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\" (UID: \"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c\") "
Oct 30 22:49:37.326306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.326285    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca" pod "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" (UID: "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:37.326529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.326512    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.328049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.328031    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj" pod "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" (UID: "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c"). InnerVolumeSpecName "kube-api-access-thggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.328159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.328145    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca" pod "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" (UID: "d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:37.365189 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.365166    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:49:37.367600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.367581    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-55edb1-openshift-default-66bb67b689-sshfb"]
Oct 30 22:49:37.427540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.427481    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\" (UniqueName: \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\") pod \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") "
Oct 30 22:49:37.427666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.427601    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\") pod \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") "
Oct 30 22:49:37.427666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.427647    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\") pod \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\" (UID: \"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244\") "
Oct 30 22:49:37.428104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.428079    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca" pod "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" (UID: "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:37.428180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.428137    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.428180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.428152    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thggj\" (UniqueName: \"kubernetes.io/projected/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c-kube-api-access-thggj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.429539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.429518    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh" pod "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" (UID: "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244"). InnerVolumeSpecName "kube-api-access-t4wmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.429707 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.429686    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca" pod "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" (UID: "0cc1f7c4-ea7d-4267-8a1a-e5ce50684244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:37.528994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.528965    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.528994 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.528991    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:37.529170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:37.529003    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wmh\" (UniqueName: \"kubernetes.io/projected/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244-kube-api-access-t4wmh\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:38.152447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.152416    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"
Oct 30 22:49:38.152447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.152432    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r" event={"ID":"d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c","Type":"ContainerDied","Data":"ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca"}
Oct 30 22:49:38.152660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.152478    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad91e6b16f789bef015ce52d10c59b3dc28899f219dbba644be578eb8cc679ca"
Oct 30 22:49:38.154656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.154631    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"
Oct 30 22:49:38.154799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.154629    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b" event={"ID":"0cc1f7c4-ea7d-4267-8a1a-e5ce50684244","Type":"ContainerDied","Data":"258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32"}
Oct 30 22:49:38.154799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:38.154756    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="258e4d0fa47e2e1323d313c9b669a2348daefaaa3065622f906c5d6fe593ef32"
Oct 30 22:49:39.019470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:39.019436    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9638907f-d147-4133-8c65-45cf5c88feca" path="/var/lib/kubelet/pods/9638907f-d147-4133-8c65-45cf5c88feca/volumes"
Oct 30 22:49:41.497611 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:41.497568    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.206340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:43.206307    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.424385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:43.424348    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.606293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:45.606257    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:49:45.768091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:45.768056    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.909329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:45.909299    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.067752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:46.067714    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.962118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:46.962083    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.048887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.048854    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"]
Oct 30 22:49:48.096019 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.095993    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a64224a-trhlq"]
Oct 30 22:49:48.103440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.103412    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.266769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.266736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.492975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.492943    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.730491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.730452    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.880327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.880242    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:48.981581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:48.981552    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"]
Oct 30 22:49:49.020025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:49.019994    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b809f7d8-4601-414c-b518-8a28e82515fb" path="/var/lib/kubelet/pods/b809f7d8-4601-414c-b518-8a28e82515fb/volumes"
Oct 30 22:49:49.030517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:49.030493    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a6900c-7sz4v"]
Oct 30 22:49:49.415902 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:49.415865    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:50.522543 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:50.522509    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:49:50.607359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.607324    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" podUID="52bd35f9-2cd4-4bc8-83da-492a94c83699" containerName="payload-pre-processing" containerID="cri-o://60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7" gracePeriod=25
Oct 30 22:49:50.871959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.871929    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:49:50.973297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.973260    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\") pod \"52bd35f9-2cd4-4bc8-83da-492a94c83699\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") "
Oct 30 22:49:50.973500 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.973343    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\" (UniqueName: \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\") pod \"52bd35f9-2cd4-4bc8-83da-492a94c83699\" (UID: \"52bd35f9-2cd4-4bc8-83da-492a94c83699\") "
Oct 30 22:49:50.973643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.973621    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume" pod "52bd35f9-2cd4-4bc8-83da-492a94c83699" (UID: "52bd35f9-2cd4-4bc8-83da-492a94c83699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:50.973763 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.973750    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd35f9-2cd4-4bc8-83da-492a94c83699-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:50.975419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:50.975397    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz" pod "52bd35f9-2cd4-4bc8-83da-492a94c83699" (UID: "52bd35f9-2cd4-4bc8-83da-492a94c83699"). InnerVolumeSpecName "kube-api-access-n42nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:51.037317 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.037233    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb62ad1-a22f-4010-ab81-c8b6d935d781" path="/var/lib/kubelet/pods/1cb62ad1-a22f-4010-ab81-c8b6d935d781/volumes"
Oct 30 22:49:51.074335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.074305    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42nz\" (UniqueName: \"kubernetes.io/projected/52bd35f9-2cd4-4bc8-83da-492a94c83699-kube-api-access-n42nz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.237397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.237362    4410 generic.go:356] "Generic (PLEG): container finished" podID="52bd35f9-2cd4-4bc8-83da-492a94c83699" containerID="60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7" exitCode=0
Oct 30 22:49:51.237616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.237418    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"
Oct 30 22:49:51.237616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.237443    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" event={"ID":"52bd35f9-2cd4-4bc8-83da-492a94c83699","Type":"ContainerDied","Data":"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"}
Oct 30 22:49:51.237616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.237508    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c" event={"ID":"52bd35f9-2cd4-4bc8-83da-492a94c83699","Type":"ContainerDied","Data":"ff49d7edbb0caacb7245d37b73a2d08034e5244875a791b4ab40c3d9497f94fb"}
Oct 30 22:49:51.237616 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.237539    4410 scope.go:122] "RemoveContainer" containerID="60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"
Oct 30 22:49:51.246980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.246964    4410 scope.go:122] "RemoveContainer" containerID="60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"
Oct 30 22:49:51.247288 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:49:51.247262    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7\": container with ID starting with 60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7 not found: ID does not exist" containerID="60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"
Oct 30 22:49:51.247339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.247296    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7"} err="failed to get container status \"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7\": rpc error: code = NotFound desc = could not find container \"60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7\": container with ID starting with 60e3254645a58b5160ba44347c9a23653342f135800fbb15b8239c6bc7abcba7 not found: ID does not exist"
Oct 30 22:49:51.401593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.401567    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:49:51.404858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:51.404836    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-15e0f6-7c65f9b455-6955c"]
Oct 30 22:49:52.454147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:52.454111    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.746270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:52.746184    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.942392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:52.942356    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"]
Oct 30 22:49:52.991217 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:52.991189    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-15e0f6-g942b"]
Oct 30 22:49:53.019176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:53.019102    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" path="/var/lib/kubelet/pods/0cc1f7c4-ea7d-4267-8a1a-e5ce50684244/volumes"
Oct 30 22:49:53.019571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:53.019550    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bd35f9-2cd4-4bc8-83da-492a94c83699" path="/var/lib/kubelet/pods/52bd35f9-2cd4-4bc8-83da-492a94c83699/volumes"
Oct 30 22:49:53.360054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:53.359973    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:53.529732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:53.529699    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.012456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:54.012419    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.336649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.336614    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.753307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.753274    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.911792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.911743    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.923050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923012    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:49:55.923804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923751    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" containerName="revoke-keys"
Oct 30 22:49:55.923804 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923790    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" containerName="revoke-keys"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923867    4410 cpu_manager.go:414] "containerMap: removing container" podUID="52bd35f9-2cd4-4bc8-83da-492a94c83699" containerName="payload-pre-processing"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923876    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bd35f9-2cd4-4bc8-83da-492a94c83699" containerName="payload-pre-processing"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923893    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923901    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923918    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923926    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923934    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" containerName="revoke-keys"
Oct 30 22:49:55.923979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.923941    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" containerName="revoke-keys"
Oct 30 22:49:55.924468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.924069    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:55.924468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.924082    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc1f7c4-ea7d-4267-8a1a-e5ce50684244" containerName="revoke-keys"
Oct 30 22:49:55.924468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.924092    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" containerName="revoke-keys"
Oct 30 22:49:55.924468 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.924113    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bd35f9-2cd4-4bc8-83da-492a94c83699" containerName="payload-pre-processing"
Oct 30 22:49:55.929046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.929023    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:55.981567 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:55.981535    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6924e5-openshift-default-dockercfg-dh87l\""
Oct 30 22:49:56.029693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.029604    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.029966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.029942    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030114    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030283 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030263    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030403    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030556 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030541    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030714    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.030924 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.030905    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkv7\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.031118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.031100    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.035079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.035053    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:49:56.066572 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.066536    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:56.131828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.131790    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.131851    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.131879    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.131925    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.131976    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132005    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132055    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132165    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132274    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkv7\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132547    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.132961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132933    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.133106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.132990    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.133313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.133283    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.133438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.133412    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.135118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.135097    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.135569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.135552    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.189469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.189435    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.189937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.189910    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkv7\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\") pod \"e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") " pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.222076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.221920    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:56.254681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.254646    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:56.687957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.687928    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:49:56.689879 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:49:56.689849    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2820b03_a73f_4f04_94c5_40adab80bbef.slice/crio-9ad378bd9191a13d5f3840eca5316215bc9ee86a197135ed2b71f69baf6d8e43 WatchSource:0}: Error finding container 9ad378bd9191a13d5f3840eca5316215bc9ee86a197135ed2b71f69baf6d8e43: Status 404 returned error can't find the container with id 9ad378bd9191a13d5f3840eca5316215bc9ee86a197135ed2b71f69baf6d8e43
Oct 30 22:49:56.698582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.698541    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:49:56.698682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.698634    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:49:56.698727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.698680    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:49:56.902740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.902704    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"]
Oct 30 22:49:56.903295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.903280    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9638907f-d147-4133-8c65-45cf5c88feca" containerName="istio-proxy"
Oct 30 22:49:56.906474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.906457    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:56.961853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.961688    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:56.961853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.961762    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:49:56.962101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.961986    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:56.962321 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.962298    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:56.965153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:56.965126    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"]
Oct 30 22:49:57.046614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.046577    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.046816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.046730    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.046901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.046827    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskmd\" (UniqueName: \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.147937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.147904    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.148091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.148044    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.148142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.148089    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskmd\" (UniqueName: \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.148691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.148665    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.150242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.150220    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.208467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.208429    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskmd\" (UniqueName: \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\") pod \"maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.220034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.219965    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:49:57.278053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.278010    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerStarted","Data":"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"}
Oct 30 22:49:57.278293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.278081    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerStarted","Data":"9ad378bd9191a13d5f3840eca5316215bc9ee86a197135ed2b71f69baf6d8e43"}
Oct 30 22:49:57.405950 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.405887    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podStartSLOduration=2.405867341 podStartE2EDuration="2.405867341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:57.402694876 +0000 UTC m=+4050.491432692" watchObservedRunningTime="2026-10-30 22:49:57.405867341 +0000 UTC m=+4050.494605145"
Oct 30 22:49:57.610163 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:57.610096    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"]
Oct 30 22:49:57.612598 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:49:57.612573    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5bf441_2006_4d65_96cd_dfbf683585ca.slice/crio-4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005 WatchSource:0}: Error finding container 4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005: Status 404 returned error can't find the container with id 4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005
Oct 30 22:49:58.254951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.254918    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:58.260667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.260644    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:58.286418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.286385    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd" event={"ID":"3d5bf441-2006-4d65-96cd-dfbf683585ca","Type":"ContainerStarted","Data":"ca727aa51f9e3c7e5cccb50e91de2333edf5e466b5210d1f09e3331f6a1a210d"}
Oct 30 22:49:58.286539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.286437    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd" event={"ID":"3d5bf441-2006-4d65-96cd-dfbf683585ca","Type":"ContainerStarted","Data":"4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005"}
Oct 30 22:49:58.286768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.286744    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:58.287768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.287748    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:49:58.610837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:58.610760    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd" podStartSLOduration=2.610746229 podStartE2EDuration="2.610746229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:58.608407423 +0000 UTC m=+4051.697145227" watchObservedRunningTime="2026-10-30 22:49:58.610746229 +0000 UTC m=+4051.699484031"
Oct 30 22:49:59.045338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.045302    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"]
Oct 30 22:49:59.047953 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.047925    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0316043a-pmggg"]
Oct 30 22:49:59.293694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.293661    4410 generic.go:356] "Generic (PLEG): container finished" podID="3d5bf441-2006-4d65-96cd-dfbf683585ca" containerID="ca727aa51f9e3c7e5cccb50e91de2333edf5e466b5210d1f09e3331f6a1a210d" exitCode=0
Oct 30 22:49:59.294115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.293727    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd" event={"ID":"3d5bf441-2006-4d65-96cd-dfbf683585ca","Type":"ContainerDied","Data":"ca727aa51f9e3c7e5cccb50e91de2333edf5e466b5210d1f09e3331f6a1a210d"}
Oct 30 22:49:59.390410 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.390373    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:59.849437 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:49:59.849350    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.045227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.045197    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.217974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.217937    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:50:00.218167 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.218143    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy" containerID="cri-o://e9b449618d784a38120a883732be24067ea3639c6550da8cf61c89fdac256a4e" gracePeriod=30
Oct 30 22:50:00.231313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.231276    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.455402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.455375    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:50:00.481983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.481931    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\") pod \"3d5bf441-2006-4d65-96cd-dfbf683585ca\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") "
Oct 30 22:50:00.481983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.481981    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\") pod \"3d5bf441-2006-4d65-96cd-dfbf683585ca\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") "
Oct 30 22:50:00.482123 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.482078    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\" (UniqueName: \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\") pod \"3d5bf441-2006-4d65-96cd-dfbf683585ca\" (UID: \"3d5bf441-2006-4d65-96cd-dfbf683585ca\") "
Oct 30 22:50:00.482265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.482242    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca" pod "3d5bf441-2006-4d65-96cd-dfbf683585ca" (UID: "3d5bf441-2006-4d65-96cd-dfbf683585ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:00.482433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.482420    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.484045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.484022    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca" pod "3d5bf441-2006-4d65-96cd-dfbf683585ca" (UID: "3d5bf441-2006-4d65-96cd-dfbf683585ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:00.484153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.484132    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd" pod "3d5bf441-2006-4d65-96cd-dfbf683585ca" (UID: "3d5bf441-2006-4d65-96cd-dfbf683585ca"). InnerVolumeSpecName "kube-api-access-gskmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:00.583495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.583467    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5bf441-2006-4d65-96cd-dfbf683585ca-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.583628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.583501    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gskmd\" (UniqueName: \"kubernetes.io/projected/3d5bf441-2006-4d65-96cd-dfbf683585ca-kube-api-access-gskmd\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.688526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:00.688497    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:01.019770 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.019735    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3a96cc-db66-4c2f-bd91-d3414a2ca44f" path="/var/lib/kubelet/pods/fa3a96cc-db66-4c2f-bd91-d3414a2ca44f/volumes"
Oct 30 22:50:01.033666 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.033430    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"]
Oct 30 22:50:01.034102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.034081    4410 cpu_manager.go:414] "containerMap: removing container" podUID="3d5bf441-2006-4d65-96cd-dfbf683585ca" containerName="revoke-keys"
Oct 30 22:50:01.034102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.034102    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5bf441-2006-4d65-96cd-dfbf683585ca" containerName="revoke-keys"
Oct 30 22:50:01.034297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.034283    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5bf441-2006-4d65-96cd-dfbf683585ca" containerName="revoke-keys"
Oct 30 22:50:01.037703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.037682    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.088496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.088460    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.088627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.088589    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.088627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.088623    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx9n\" (UniqueName: \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.097463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.097431    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"]
Oct 30 22:50:01.189631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.189602    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.189760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.189639    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx9n\" (UniqueName: \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.189760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.189729    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.190260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.190239    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.192084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.192066    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.246978 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.246958    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx9n\" (UniqueName: \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\") pod \"maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.309982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.309931    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"
Oct 30 22:50:01.309982 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.309949    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd" event={"ID":"3d5bf441-2006-4d65-96cd-dfbf683585ca","Type":"ContainerDied","Data":"4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005"}
Oct 30 22:50:01.310107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.309994    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f51d7f695b70629df320fdd42dfcae2b2d7e931888b915dba86e77ebbe40005"
Oct 30 22:50:01.357527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.357506    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:01.731261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:01.731233    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"]
Oct 30 22:50:01.732537 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:50:01.732506    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eae7d0d_d2a0_4b89_9e3a_83b04b019403.slice/crio-f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f WatchSource:0}: Error finding container f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f: Status 404 returned error can't find the container with id f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f
Oct 30 22:50:02.316896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:02.316809    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs" event={"ID":"9eae7d0d-d2a0-4b89-9e3a-83b04b019403","Type":"ContainerStarted","Data":"ebd39341ed0704ca00b91064ae79644928abe79ab417183bae82029cb01a48cb"}
Oct 30 22:50:02.316896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:02.316872    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs" event={"ID":"9eae7d0d-d2a0-4b89-9e3a-83b04b019403","Type":"ContainerStarted","Data":"f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f"}
Oct 30 22:50:02.433745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:02.433695    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs" podStartSLOduration=2.433683146 podStartE2EDuration="2.433683146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:02.431603263 +0000 UTC m=+4055.520341065" watchObservedRunningTime="2026-10-30 22:50:02.433683146 +0000 UTC m=+4055.522420948"
Oct 30 22:50:03.324441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:03.324403    4410 generic.go:356] "Generic (PLEG): container finished" podID="9eae7d0d-d2a0-4b89-9e3a-83b04b019403" containerID="ebd39341ed0704ca00b91064ae79644928abe79ab417183bae82029cb01a48cb" exitCode=0
Oct 30 22:50:03.324901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:03.324453    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs" event={"ID":"9eae7d0d-d2a0-4b89-9e3a-83b04b019403","Type":"ContainerDied","Data":"ebd39341ed0704ca00b91064ae79644928abe79ab417183bae82029cb01a48cb"}
Oct 30 22:50:04.013591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.013558    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:04.468612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.468582    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:04.524672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.524644    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\" (UniqueName: \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\") pod \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") "
Oct 30 22:50:04.524872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.524858    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\") pod \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") "
Oct 30 22:50:04.524923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.524896    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\") pod \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\" (UID: \"9eae7d0d-d2a0-4b89-9e3a-83b04b019403\") "
Oct 30 22:50:04.525335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.525308    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca" pod "9eae7d0d-d2a0-4b89-9e3a-83b04b019403" (UID: "9eae7d0d-d2a0-4b89-9e3a-83b04b019403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:04.526816 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.526790    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n" pod "9eae7d0d-d2a0-4b89-9e3a-83b04b019403" (UID: "9eae7d0d-d2a0-4b89-9e3a-83b04b019403"). InnerVolumeSpecName "kube-api-access-4xx9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:04.526927 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.526859    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca" pod "9eae7d0d-d2a0-4b89-9e3a-83b04b019403" (UID: "9eae7d0d-d2a0-4b89-9e3a-83b04b019403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:04.626203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.626137    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.626203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.626164    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.626203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:04.626176    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xx9n\" (UniqueName: \"kubernetes.io/projected/9eae7d0d-d2a0-4b89-9e3a-83b04b019403-kube-api-access-4xx9n\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.338442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.338416    4410 generic.go:356] "Generic (PLEG): container finished" podID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerID="e9b449618d784a38120a883732be24067ea3639c6550da8cf61c89fdac256a4e" exitCode=0
Oct 30 22:50:05.338521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.338492    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerDied","Data":"e9b449618d784a38120a883732be24067ea3639c6550da8cf61c89fdac256a4e"}
Oct 30 22:50:05.338570 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.338557    4410 scope.go:122] "RemoveContainer" containerID="b8d3d76d53b291060ef07678a3a514ff6282732826171e8dc08fefee735ad894"
Oct 30 22:50:05.340218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.340201    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"
Oct 30 22:50:05.340332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.340221    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs" event={"ID":"9eae7d0d-d2a0-4b89-9e3a-83b04b019403","Type":"ContainerDied","Data":"f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f"}
Oct 30 22:50:05.340332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.340274    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46933e969bc04594656f53f358e94bd699b7e8039fcde807fc0ed66f119654f"
Oct 30 22:50:05.490681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.490653    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:50:05.538566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538541    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538589    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538630    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538653    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538694 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538676    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538697    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538833    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:05.538961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538857    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538897    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.538961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538957    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\") pod \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\" (UID: \"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a\") "
Oct 30 22:50:05.539253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.538990    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:05.539253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539010    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:05.539253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539099    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:05.539253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539178    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:05.539501 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539477    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.539573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539507    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.539573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539522    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.539573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539536    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.539573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.539549    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.541085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.541055    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:05.541085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.541074    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:05.541257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.541084    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:05.541257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.541110    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv" pod "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" (UID: "0c9b0fc7-7788-4ad5-b000-8a46c2964a2a"). InnerVolumeSpecName "kube-api-access-bd4hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:05.561723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.561700    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.640422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.640390    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.640422 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.640417    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.640640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.640429    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4hv\" (UniqueName: \"kubernetes.io/projected/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-kube-api-access-bd4hv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.640640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.640439    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:05.821202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.821156    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.977589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:05.977497    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.128823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.128795    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"]
Oct 30 22:50:06.131199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.131178    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20ccef-4ndzv"]
Oct 30 22:50:06.347541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.347463    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"
Oct 30 22:50:06.347541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.347495    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8" event={"ID":"0c9b0fc7-7788-4ad5-b000-8a46c2964a2a","Type":"ContainerDied","Data":"1e978992d25ba6549384cb7573449ac2ca0dceb809703e7d51069e5dee8c69c7"}
Oct 30 22:50:06.347738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.347552    4410 scope.go:122] "RemoveContainer" containerID="e9b449618d784a38120a883732be24067ea3639c6550da8cf61c89fdac256a4e"
Oct 30 22:50:06.491879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.491843    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.510863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.510832    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:50:06.560103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:06.560077    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-15e0f6-openshift-default-6fbd5577b9-xztj8"]
Oct 30 22:50:07.022151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:07.022112    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" path="/var/lib/kubelet/pods/0c9b0fc7-7788-4ad5-b000-8a46c2964a2a/volumes"
Oct 30 22:50:07.022700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:07.022685    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba79a582-5b82-46bb-b993-2d847b808631" path="/var/lib/kubelet/pods/ba79a582-5b82-46bb-b993-2d847b808631/volumes"
Oct 30 22:50:07.359970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:07.359890    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:07.567377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:07.567333    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:08.961746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:08.961708    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:09.249834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:09.249720    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.194882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:10.194848    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.425529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:10.425493    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.588544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:10.588461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.926102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:10.926073    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:12.561117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:12.561085    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:50:12.672155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:12.672123    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:50:12.947533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:12.947501    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:13.169395 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:13.169362    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:13.410682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:13.410643    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:14.171285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171254    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"]
Oct 30 22:50:14.171758 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171745    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:14.171841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171759    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:14.171841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171788    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:14.171841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171796    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:14.171841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171823    4410 cpu_manager.go:414] "containerMap: removing container" podUID="9eae7d0d-d2a0-4b89-9e3a-83b04b019403" containerName="revoke-keys"
Oct 30 22:50:14.171841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171831    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eae7d0d-d2a0-4b89-9e3a-83b04b019403" containerName="revoke-keys"
Oct 30 22:50:14.172010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171926    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:14.172010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.171938    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eae7d0d-d2a0-4b89-9e3a-83b04b019403" containerName="revoke-keys"
Oct 30 22:50:14.176296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.176278    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.229408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.229376    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.229576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.229499    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.229644 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.229627    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97plc\" (UniqueName: \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.232768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.232743    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"]
Oct 30 22:50:14.278630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.278596    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:14.278831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.278650    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:14.278831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.278661    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:14.278831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.278718    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:50:14.330757 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.330724    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.330937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.330805    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97plc\" (UniqueName: \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.330937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.330871    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.331486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.331461    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.333090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.333068    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.387791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.387745    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97plc\" (UniqueName: \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\") pod \"maas-api-revoke-keys-e2e-extmod-b-32af64-t4699\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.488834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.488746    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:14.861519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:14.861494    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"]
Oct 30 22:50:14.863763 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:50:14.863736    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe39ce5_e42c_46c8_91ca_57c1975a8c82.slice/crio-8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564 WatchSource:0}: Error finding container 8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564: Status 404 returned error can't find the container with id 8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564
Oct 30 22:50:15.404460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:15.404370    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699" event={"ID":"0fe39ce5-e42c-46c8-91ca-57c1975a8c82","Type":"ContainerStarted","Data":"5e37501505c8c5550712e8866fe756b1952f44e8b64e2c1e3c171fbc418b3015"}
Oct 30 22:50:15.404460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:15.404426    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699" event={"ID":"0fe39ce5-e42c-46c8-91ca-57c1975a8c82","Type":"ContainerStarted","Data":"8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564"}
Oct 30 22:50:15.517828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:15.517754    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699" podStartSLOduration=1.5177397670000001 podStartE2EDuration="1.517739767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:15.515615868 +0000 UTC m=+4068.604353670" watchObservedRunningTime="2026-10-30 22:50:15.517739767 +0000 UTC m=+4068.606477570"
Oct 30 22:50:16.412945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:16.412907    4410 generic.go:356] "Generic (PLEG): container finished" podID="0fe39ce5-e42c-46c8-91ca-57c1975a8c82" containerID="5e37501505c8c5550712e8866fe756b1952f44e8b64e2c1e3c171fbc418b3015" exitCode=0
Oct 30 22:50:16.412945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:16.412941    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699" event={"ID":"0fe39ce5-e42c-46c8-91ca-57c1975a8c82","Type":"ContainerDied","Data":"5e37501505c8c5550712e8866fe756b1952f44e8b64e2c1e3c171fbc418b3015"}
Oct 30 22:50:16.421103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:16.421072    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.561920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.561886    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" podUID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" containerName="payload-processing" containerID="cri-o://a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf" gracePeriod=25
Oct 30 22:50:17.653729 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.653704    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:17.672802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.672719    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" podUID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" containerName="payload-pre-processing" containerID="cri-o://5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41" gracePeriod=25
Oct 30 22:50:17.699431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.699400    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.770811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.770517    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\") pod \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") "
Oct 30 22:50:17.770811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.770584    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\") pod \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") "
Oct 30 22:50:17.771117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.770951    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca" pod "0fe39ce5-e42c-46c8-91ca-57c1975a8c82" (UID: "0fe39ce5-e42c-46c8-91ca-57c1975a8c82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:17.771715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.770764    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\" (UniqueName: \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\") pod \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\" (UID: \"0fe39ce5-e42c-46c8-91ca-57c1975a8c82\") "
Oct 30 22:50:17.771715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.771684    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:17.773737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.773704    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc" pod "0fe39ce5-e42c-46c8-91ca-57c1975a8c82" (UID: "0fe39ce5-e42c-46c8-91ca-57c1975a8c82"). InnerVolumeSpecName "kube-api-access-97plc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.775552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.775519    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca" pod "0fe39ce5-e42c-46c8-91ca-57c1975a8c82" (UID: "0fe39ce5-e42c-46c8-91ca-57c1975a8c82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:17.855378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.855337    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:50:17.872749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.872722    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97plc\" (UniqueName: \"kubernetes.io/projected/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-kube-api-access-97plc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:17.872749 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.872749    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe39ce5-e42c-46c8-91ca-57c1975a8c82-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:17.887119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.887081    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.973426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.973251    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:50:17.977722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.977688    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\" (UniqueName: \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\") pod \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") "
Oct 30 22:50:17.977952 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.977930    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\") pod \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\" (UID: \"1a41aea1-38f5-4645-8f3c-5e01bbdabac3\") "
Oct 30 22:50:17.978455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.978418    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume" pod "1a41aea1-38f5-4645-8f3c-5e01bbdabac3" (UID: "1a41aea1-38f5-4645-8f3c-5e01bbdabac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:17.978925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:17.978897    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr" pod "1a41aea1-38f5-4645-8f3c-5e01bbdabac3" (UID: "1a41aea1-38f5-4645-8f3c-5e01bbdabac3"). InnerVolumeSpecName "kube-api-access-crbxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:18.078766 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.078734    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\" (UniqueName: \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\") pod \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") "
Oct 30 22:50:18.079013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.078910    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\") pod \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\" (UID: \"56a21e6b-00df-4c92-b1c7-0fc725fefcbc\") "
Oct 30 22:50:18.079470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.079446    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:18.079569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.079480    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbxr\" (UniqueName: \"kubernetes.io/projected/1a41aea1-38f5-4645-8f3c-5e01bbdabac3-kube-api-access-crbxr\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:18.079569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.079451    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume" pod "56a21e6b-00df-4c92-b1c7-0fc725fefcbc" (UID: "56a21e6b-00df-4c92-b1c7-0fc725fefcbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:18.080869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.080848    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7" pod "56a21e6b-00df-4c92-b1c7-0fc725fefcbc" (UID: "56a21e6b-00df-4c92-b1c7-0fc725fefcbc"). InnerVolumeSpecName "kube-api-access-8vwm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:18.181416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.181379    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:18.181416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.181414    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwm7\" (UniqueName: \"kubernetes.io/projected/56a21e6b-00df-4c92-b1c7-0fc725fefcbc-kube-api-access-8vwm7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:18.425604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.425569    4410 generic.go:356] "Generic (PLEG): container finished" podID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" containerID="a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf" exitCode=0
Oct 30 22:50:18.425827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.425614    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"
Oct 30 22:50:18.425827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.425651    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" event={"ID":"1a41aea1-38f5-4645-8f3c-5e01bbdabac3","Type":"ContainerDied","Data":"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"}
Oct 30 22:50:18.425827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.425701    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9" event={"ID":"1a41aea1-38f5-4645-8f3c-5e01bbdabac3","Type":"ContainerDied","Data":"dac81e9d3fb0ba9465979687032ac80c4ee8f47ac19ace3967625b1fe397f958"}
Oct 30 22:50:18.425827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.425729    4410 scope.go:122] "RemoveContainer" containerID="a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"
Oct 30 22:50:18.427180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.427158    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"
Oct 30 22:50:18.427322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.427156    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699" event={"ID":"0fe39ce5-e42c-46c8-91ca-57c1975a8c82","Type":"ContainerDied","Data":"8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564"}
Oct 30 22:50:18.427322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.427284    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ebeed62677f4b68de73a54fbfe3762937375fcae9094c8544ec2660188ff564"
Oct 30 22:50:18.428604 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.428579    4410 generic.go:356] "Generic (PLEG): container finished" podID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" containerID="5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41" exitCode=0
Oct 30 22:50:18.428721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.428623    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"
Oct 30 22:50:18.428721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.428688    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" event={"ID":"56a21e6b-00df-4c92-b1c7-0fc725fefcbc","Type":"ContainerDied","Data":"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"}
Oct 30 22:50:18.428883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.428730    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd" event={"ID":"56a21e6b-00df-4c92-b1c7-0fc725fefcbc","Type":"ContainerDied","Data":"0b26caa3f3d4ff583b917cca71f106c7dc3b981259cb65d48416f7f7423a8b6f"}
Oct 30 22:50:18.436883 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.436862    4410 scope.go:122] "RemoveContainer" containerID="a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"
Oct 30 22:50:18.437484 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:18.437464    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf\": container with ID starting with a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf not found: ID does not exist" containerID="a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"
Oct 30 22:50:18.437579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.437495    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf"} err="failed to get container status \"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf\": rpc error: code = NotFound desc = could not find container \"a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf\": container with ID starting with a1d56564613ed6d2a467833fa1401b7a50b3ca1dedcfbb84abd53bc905603faf not found: ID does not exist"
Oct 30 22:50:18.437579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.437518    4410 scope.go:122] "RemoveContainer" containerID="5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"
Oct 30 22:50:18.446573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.446556    4410 scope.go:122] "RemoveContainer" containerID="5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"
Oct 30 22:50:18.446818 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:18.446800    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41\": container with ID starting with 5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41 not found: ID does not exist" containerID="5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"
Oct 30 22:50:18.446865 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.446824    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41"} err="failed to get container status \"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41\": rpc error: code = NotFound desc = could not find container \"5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41\": container with ID starting with 5878df8596783adaa7abb6eaa2e1ed6efd56d37d0cdd707b7d507eff4f63ab41 not found: ID does not exist"
Oct 30 22:50:18.588652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.588623    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:50:18.592913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.592884    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b1b1eb-5cbf6cf4bc-glsx9"]
Oct 30 22:50:18.701126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.701058    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:50:18.704619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:18.704593    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b1b1eb-7f6b9c9d9f-7zxkd"]
Oct 30 22:50:19.019893 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:19.019816    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" path="/var/lib/kubelet/pods/1a41aea1-38f5-4645-8f3c-5e01bbdabac3/volumes"
Oct 30 22:50:19.020225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:19.020211    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" path="/var/lib/kubelet/pods/56a21e6b-00df-4c92-b1c7-0fc725fefcbc/volumes"
Oct 30 22:50:20.521444 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:20.521408    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:50:21.420545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:21.419539    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:25.676810 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:25.676762    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:50:25.781392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:25.781363    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:50:26.065346 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:26.065271    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:26.231024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:26.230983    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:30.677713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:30.677659    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" podUID="7891b92e-6d14-4644-ba7e-f2123858d752" containerName="payload-processing" containerID="cri-o://fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5" gracePeriod=25
Oct 30 22:50:30.782003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:30.781964    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" podUID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" containerName="payload-pre-processing" containerID="cri-o://02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee" gracePeriod=25
Oct 30 22:50:30.945743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:30.945683    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:50:31.020946 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.020910    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:31.058538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.058510    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:50:31.114397 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.114358    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\" (UniqueName: \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\") pod \"7891b92e-6d14-4644-ba7e-f2123858d752\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") "
Oct 30 22:50:31.114600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.114443    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\") pod \"7891b92e-6d14-4644-ba7e-f2123858d752\" (UID: \"7891b92e-6d14-4644-ba7e-f2123858d752\") "
Oct 30 22:50:31.115639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.115602    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume" pod "7891b92e-6d14-4644-ba7e-f2123858d752" (UID: "7891b92e-6d14-4644-ba7e-f2123858d752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:31.122951 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.122920    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz" pod "7891b92e-6d14-4644-ba7e-f2123858d752" (UID: "7891b92e-6d14-4644-ba7e-f2123858d752"). InnerVolumeSpecName "kube-api-access-w8whz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:31.216128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216059    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\") pod \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") "
Oct 30 22:50:31.216287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216269    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\" (UniqueName: \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\") pod \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\" (UID: \"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306\") "
Oct 30 22:50:31.216470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216450    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume" pod "87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" (UID: "87d7cfd1-cb7c-452f-876f-2fbc6ffaa306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:31.216627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216613    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8whz\" (UniqueName: \"kubernetes.io/projected/7891b92e-6d14-4644-ba7e-f2123858d752-kube-api-access-w8whz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:31.216663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216630    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:31.216663 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.216639    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7891b92e-6d14-4644-ba7e-f2123858d752-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:31.218243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.218218    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b" pod "87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" (UID: "87d7cfd1-cb7c-452f-876f-2fbc6ffaa306"). InnerVolumeSpecName "kube-api-access-vt94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:31.317854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.317831    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt94b\" (UniqueName: \"kubernetes.io/projected/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306-kube-api-access-vt94b\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:31.508199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.508127    4410 generic.go:356] "Generic (PLEG): container finished" podID="7891b92e-6d14-4644-ba7e-f2123858d752" containerID="fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5" exitCode=0
Oct 30 22:50:31.508199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.508184    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"
Oct 30 22:50:31.508393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.508216    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" event={"ID":"7891b92e-6d14-4644-ba7e-f2123858d752","Type":"ContainerDied","Data":"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"}
Oct 30 22:50:31.508393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.508299    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4" event={"ID":"7891b92e-6d14-4644-ba7e-f2123858d752","Type":"ContainerDied","Data":"0730ffcc8d4f2483474b72f961ca3401333ad847cbe20e26872cc58bae316b1b"}
Oct 30 22:50:31.508393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.508328    4410 scope.go:122] "RemoveContainer" containerID="fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"
Oct 30 22:50:31.509721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.509696    4410 generic.go:356] "Generic (PLEG): container finished" podID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" containerID="02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee" exitCode=0
Oct 30 22:50:31.509868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.509745    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"
Oct 30 22:50:31.509868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.509811    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" event={"ID":"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306","Type":"ContainerDied","Data":"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"}
Oct 30 22:50:31.509868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.509857    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm" event={"ID":"87d7cfd1-cb7c-452f-876f-2fbc6ffaa306","Type":"ContainerDied","Data":"4fd19e36f3e184b820727412aa0a41677d20dfdc76f370c0727b2b3e0d7f0476"}
Oct 30 22:50:31.519316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.519297    4410 scope.go:122] "RemoveContainer" containerID="fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"
Oct 30 22:50:31.519600 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:31.519578    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5\": container with ID starting with fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5 not found: ID does not exist" containerID="fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"
Oct 30 22:50:31.519673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.519607    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5"} err="failed to get container status \"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5\": rpc error: code = NotFound desc = could not find container \"fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5\": container with ID starting with fee32c9841bfd726f2a4eefc5d584b0385697b75959bbb5813beda59933fc7f5 not found: ID does not exist"
Oct 30 22:50:31.519673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.519625    4410 scope.go:122] "RemoveContainer" containerID="02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"
Oct 30 22:50:31.529656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.529637    4410 scope.go:122] "RemoveContainer" containerID="02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"
Oct 30 22:50:31.529938 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:31.529918    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee\": container with ID starting with 02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee not found: ID does not exist" containerID="02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"
Oct 30 22:50:31.529999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.529944    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee"} err="failed to get container status \"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee\": rpc error: code = NotFound desc = could not find container \"02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee\": container with ID starting with 02b2e0e31f182a44895dc8061f8877f6a89a35468870fb8cfbc75e2dcdd1acee not found: ID does not exist"
Oct 30 22:50:31.674392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.674362    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:50:31.722811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.722784    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-32af64-56bd55d965-49qp4"]
Oct 30 22:50:31.789408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.789345    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:50:31.794448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:31.794425    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-32af64-65f8bcc879-vdpvm"]
Oct 30 22:50:33.019297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:33.019254    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7891b92e-6d14-4644-ba7e-f2123858d752" path="/var/lib/kubelet/pods/7891b92e-6d14-4644-ba7e-f2123858d752/volumes"
Oct 30 22:50:33.019868 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:33.019847    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" path="/var/lib/kubelet/pods/87d7cfd1-cb7c-452f-876f-2fbc6ffaa306/volumes"
Oct 30 22:50:35.895710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:35.895676    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:36.943273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:36.943238    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:40.071295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:40.071211    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:41.697648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:41.697613    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:41.881685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:41.881644    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:42.064053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:42.063967    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:42.138430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:42.138397    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"]
Oct 30 22:50:42.187088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:42.187056    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-32af64-t4699"]
Oct 30 22:50:42.620045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:42.620003    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.018843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:43.018812    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe39ce5-e42c-46c8-91ca-57c1975a8c82" path="/var/lib/kubelet/pods/0fe39ce5-e42c-46c8-91ca-57c1975a8c82/volumes"
Oct 30 22:50:46.399060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:46.399029    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.047273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.047239    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.263481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.263448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.426288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.426255    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.557156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.557123    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"]
Oct 30 22:50:48.606446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.606417    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4deef-g6p8r"]
Oct 30 22:50:48.713922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.713849    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.932544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:48.932510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:49.020032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.019950    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c" path="/var/lib/kubelet/pods/d8ae1a4d-b683-4bfa-a4db-72f9e5fe424c/volumes"
Oct 30 22:50:49.105053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.104994    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:49.918689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.918652    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"]
Oct 30 22:50:49.919223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919203    4410 cpu_manager.go:414] "containerMap: removing container" podUID="7891b92e-6d14-4644-ba7e-f2123858d752" containerName="payload-processing"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919225    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7891b92e-6d14-4644-ba7e-f2123858d752" containerName="payload-processing"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919256    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0fe39ce5-e42c-46c8-91ca-57c1975a8c82" containerName="revoke-keys"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919265    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe39ce5-e42c-46c8-91ca-57c1975a8c82" containerName="revoke-keys"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919290    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" containerName="payload-processing"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919298    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" containerName="payload-processing"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919308    4410 cpu_manager.go:414] "containerMap: removing container" podUID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" containerName="payload-pre-processing"
Oct 30 22:50:49.919340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919315    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" containerName="payload-pre-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919360    4410 cpu_manager.go:414] "containerMap: removing container" podUID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" containerName="payload-pre-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919370    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" containerName="payload-pre-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919490    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe39ce5-e42c-46c8-91ca-57c1975a8c82" containerName="revoke-keys"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919508    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a41aea1-38f5-4645-8f3c-5e01bbdabac3" containerName="payload-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919520    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a21e6b-00df-4c92-b1c7-0fc725fefcbc" containerName="payload-pre-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919530    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9b0fc7-7788-4ad5-b000-8a46c2964a2a" containerName="istio-proxy"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919543    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d7cfd1-cb7c-452f-876f-2fbc6ffaa306" containerName="payload-pre-processing"
Oct 30 22:50:49.919722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.919556    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="7891b92e-6d14-4644-ba7e-f2123858d752" containerName="payload-processing"
Oct 30 22:50:49.923789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.923753    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:49.980067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:49.980033    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"]
Oct 30 22:50:50.001557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.001529    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbj6s\" (UniqueName: \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.001711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.001573    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.001806 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.001747    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.026406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.026380    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:50.026739 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.026721    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:50.075682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.075656    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:50.075682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.075665    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:50:50.102686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.102661    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.102829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.102763    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbj6s\" (UniqueName: \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.102829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.102813    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.103359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.103329    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.105008 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.104989    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.159299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.159277    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbj6s\" (UniqueName: \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.236291 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.236238    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:50.521815 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:50:50.521698    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:50:50.608497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:50.608473    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"]
Oct 30 22:50:50.610271 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:50:50.610245    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5567c83_5236_4c5b_8b4e_83122149c936.slice/crio-e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e WatchSource:0}: Error finding container e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e: Status 404 returned error can't find the container with id e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e
Oct 30 22:50:51.623555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:51.623514    4410 generic.go:356] "Generic (PLEG): container finished" podID="b5567c83-5236-4c5b-8b4e-83122149c936" containerID="c3aa9ff3e836db9b160524e438417dbfe93fb749663fd9c6d289dee41311cff8" exitCode=0
Oct 30 22:50:51.623991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:51.623598    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb" event={"ID":"b5567c83-5236-4c5b-8b4e-83122149c936","Type":"ContainerDied","Data":"c3aa9ff3e836db9b160524e438417dbfe93fb749663fd9c6d289dee41311cff8"}
Oct 30 22:50:51.623991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:51.623660    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb" event={"ID":"b5567c83-5236-4c5b-8b4e-83122149c936","Type":"ContainerStarted","Data":"e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e"}
Oct 30 22:50:52.775681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.775656    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:52.827048 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.827021    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\") pod \"b5567c83-5236-4c5b-8b4e-83122149c936\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") "
Oct 30 22:50:52.827174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.827102    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\" (UniqueName: \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\") pod \"b5567c83-5236-4c5b-8b4e-83122149c936\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") "
Oct 30 22:50:52.827174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.827126    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\") pod \"b5567c83-5236-4c5b-8b4e-83122149c936\" (UID: \"b5567c83-5236-4c5b-8b4e-83122149c936\") "
Oct 30 22:50:52.827517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.827491    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca" pod "b5567c83-5236-4c5b-8b4e-83122149c936" (UID: "b5567c83-5236-4c5b-8b4e-83122149c936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:52.829024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.828990    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca" pod "b5567c83-5236-4c5b-8b4e-83122149c936" (UID: "b5567c83-5236-4c5b-8b4e-83122149c936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:52.829146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.829105    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s" pod "b5567c83-5236-4c5b-8b4e-83122149c936" (UID: "b5567c83-5236-4c5b-8b4e-83122149c936"). InnerVolumeSpecName "kube-api-access-gbj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:52.929072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.929037    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:52.929072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.929069    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbj6s\" (UniqueName: \"kubernetes.io/projected/b5567c83-5236-4c5b-8b4e-83122149c936-kube-api-access-gbj6s\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:52.929253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:52.929085    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5567c83-5236-4c5b-8b4e-83122149c936-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:53.536329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:53.536298    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:53.637164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:53.637133    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"
Oct 30 22:50:53.637342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:53.637157    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb" event={"ID":"b5567c83-5236-4c5b-8b4e-83122149c936","Type":"ContainerDied","Data":"e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e"}
Oct 30 22:50:53.637342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:53.637214    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ebb4548c4c1d61040327c674b8fcc46bdad26fcc7248167be59c907d6fca3e"
Oct 30 22:50:55.407092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:55.407056    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:55.613006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:55.612970    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:55.776083 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:55.776010    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:56.531834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.531794    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"]
Oct 30 22:50:56.532266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.532251    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b5567c83-5236-4c5b-8b4e-83122149c936" containerName="revoke-keys"
Oct 30 22:50:56.532328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.532267    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5567c83-5236-4c5b-8b4e-83122149c936" containerName="revoke-keys"
Oct 30 22:50:56.532365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.532352    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5567c83-5236-4c5b-8b4e-83122149c936" containerName="revoke-keys"
Oct 30 22:50:56.535687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.535669    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.638325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.638294    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:56.638325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.638314    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:56.638325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.638300    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:50:56.638594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.638487    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:56.642279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.642256    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"]
Oct 30 22:50:56.667751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.667722    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4j29\" (UniqueName: \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.667895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.667762    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.668013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.667993    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.769121 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.769090    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.769237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.769135    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4j29\" (UniqueName: \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.769237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.769157    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.769706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.769683    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.771490 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.771468    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.826421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.826358    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4j29\" (UniqueName: \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:56.848986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:56.848966    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:57.228268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:57.228241    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"]
Oct 30 22:50:57.230295 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:50:57.230265    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63a2a21_0cf8_45eb_8143_c6f6b41960ad.slice/crio-792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899 WatchSource:0}: Error finding container 792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899: Status 404 returned error can't find the container with id 792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899
Oct 30 22:50:57.661983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:57.661946    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84" event={"ID":"c63a2a21-0cf8-45eb-8143-c6f6b41960ad","Type":"ContainerStarted","Data":"180ec93c56a566df5520c97ee3ab821d573ae45e64d6e6523b3031c420bfe5cf"}
Oct 30 22:50:57.662364 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:57.662000    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84" event={"ID":"c63a2a21-0cf8-45eb-8143-c6f6b41960ad","Type":"ContainerStarted","Data":"792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899"}
Oct 30 22:50:57.776560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:57.776500    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84" podStartSLOduration=1.776482466 podStartE2EDuration="1.776482466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:57.773957012 +0000 UTC m=+4110.862694816" watchObservedRunningTime="2026-10-30 22:50:57.776482466 +0000 UTC m=+4110.865220269"
Oct 30 22:50:58.671516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:58.671478    4410 generic.go:356] "Generic (PLEG): container finished" podID="c63a2a21-0cf8-45eb-8143-c6f6b41960ad" containerID="180ec93c56a566df5520c97ee3ab821d573ae45e64d6e6523b3031c420bfe5cf" exitCode=0
Oct 30 22:50:58.671961 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:58.671530    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84" event={"ID":"c63a2a21-0cf8-45eb-8143-c6f6b41960ad","Type":"ContainerDied","Data":"180ec93c56a566df5520c97ee3ab821d573ae45e64d6e6523b3031c420bfe5cf"}
Oct 30 22:50:58.889844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:58.889809    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:59.817039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.817013    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:50:59.898693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.898661    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\") pod \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") "
Oct 30 22:50:59.898903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.898723    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\") pod \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") "
Oct 30 22:50:59.898903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.898751    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\" (UniqueName: \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\") pod \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\" (UID: \"c63a2a21-0cf8-45eb-8143-c6f6b41960ad\") "
Oct 30 22:50:59.899050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.899030    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca" pod "c63a2a21-0cf8-45eb-8143-c6f6b41960ad" (UID: "c63a2a21-0cf8-45eb-8143-c6f6b41960ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:59.899169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.899155    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:59.900919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.900888    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca" pod "c63a2a21-0cf8-45eb-8143-c6f6b41960ad" (UID: "c63a2a21-0cf8-45eb-8143-c6f6b41960ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:59.901025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:50:59.900898    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29" pod "c63a2a21-0cf8-45eb-8143-c6f6b41960ad" (UID: "c63a2a21-0cf8-45eb-8143-c6f6b41960ad"). InnerVolumeSpecName "kube-api-access-z4j29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:00.000274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:00.000193    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:00.000274 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:00.000224    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4j29\" (UniqueName: \"kubernetes.io/projected/c63a2a21-0cf8-45eb-8143-c6f6b41960ad-kube-api-access-z4j29\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:00.683313 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:00.683286    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"
Oct 30 22:51:00.683520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:00.683285    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84" event={"ID":"c63a2a21-0cf8-45eb-8143-c6f6b41960ad","Type":"ContainerDied","Data":"792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899"}
Oct 30 22:51:00.683520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:00.683409    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792a896a230f9a8370848fceea1c4ce3a17cb4e6f2c8b5505d68e25be3a28899"
Oct 30 22:51:01.649096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:01.649062    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:51:01.929517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:01.929422    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:02.129479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:02.129435    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:04.053354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:04.053313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:06.650229 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:06.650156    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing" containerID="cri-o://4312a5fd83b5a495b18305edf15f2066de2d4668b118c3b890696385d7bbb0a7" gracePeriod=25
Oct 30 22:51:08.043576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:08.043538    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:51:08.315807 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:08.315702    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.478789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:08.478745    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.645183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:08.645149    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:09.491529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:09.491485    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.000964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.000929    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.236343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.236305    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.404610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.404575    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.475197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.475163    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"]
Oct 30 22:51:11.524496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.524463    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b1b1eb-zb2cs"]
Oct 30 22:51:11.872481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:11.872398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:13.019755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:13.019720    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eae7d0d-d2a0-4b89-9e3a-83b04b019403" path="/var/lib/kubelet/pods/9eae7d0d-d2a0-4b89-9e3a-83b04b019403/volumes"
Oct 30 22:51:13.043971 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:13.043940    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing" containerID="cri-o://f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6" gracePeriod=25
Oct 30 22:51:14.676327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:14.676296    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:14.875467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:14.875423    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.145524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:15.145494    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.357052 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:15.357006    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"]
Oct 30 22:51:15.404421 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:15.404350    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f7654-9st84"]
Oct 30 22:51:15.717420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:15.717328    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.923411 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:15.923352    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.020134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.020085    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63a2a21-0cf8-45eb-8143-c6f6b41960ad" path="/var/lib/kubelet/pods/c63a2a21-0cf8-45eb-8143-c6f6b41960ad/volumes"
Oct 30 22:51:17.425759 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.425722    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.634904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.634870    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.807389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.807295    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:17.909623 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.909580    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"]
Oct 30 22:51:17.957032 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.956997    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69058c-bvjfd"]
Oct 30 22:51:17.975898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:17.975849    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:18.316263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:18.316213    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:18.492423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:18.492384    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:18.654981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:18.654951    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:19.019398 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.019320    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5bf441-2006-4d65-96cd-dfbf683585ca" path="/var/lib/kubelet/pods/3d5bf441-2006-4d65-96cd-dfbf683585ca/volumes"
Oct 30 22:51:19.522037 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.522001    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"]
Oct 30 22:51:19.522506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.522449    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c63a2a21-0cf8-45eb-8143-c6f6b41960ad" containerName="revoke-keys"
Oct 30 22:51:19.522506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.522461    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63a2a21-0cf8-45eb-8143-c6f6b41960ad" containerName="revoke-keys"
Oct 30 22:51:19.522589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.522548    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63a2a21-0cf8-45eb-8143-c6f6b41960ad" containerName="revoke-keys"
Oct 30 22:51:19.526471 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.526454    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.586359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.586331    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:51:19.587408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.587374    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:19.587593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.587456    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:19.607684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.607654    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.607858 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.607693    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzk2\" (UniqueName: \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.607930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.607869    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.635597 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.635566    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:19.637967 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.637943    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"]
Oct 30 22:51:19.709017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.708981    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.709237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.709086    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.709237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.709118    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzk2\" (UniqueName: \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.710013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.709986    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.711529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.711501    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.766087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.766044    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzk2\" (UniqueName: \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:19.839337 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:19.839262    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:20.136114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:20.136039    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:20.217605 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:20.217580    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"]
Oct 30 22:51:20.220065 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:51:20.220034    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6dfe9_57ad_428d_ac53_beb14b79a195.slice/crio-2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c WatchSource:0}: Error finding container 2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c: Status 404 returned error can't find the container with id 2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c
Oct 30 22:51:20.521756 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:20.521722    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:51:20.806814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:20.806696    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss" event={"ID":"85f6dfe9-57ad-428d-ac53-beb14b79a195","Type":"ContainerStarted","Data":"72b064a77c564f272739d783aabdbd77c735d3c9f6a8a7527c9adb6706d7c979"}
Oct 30 22:51:20.806814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:20.806762    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss" event={"ID":"85f6dfe9-57ad-428d-ac53-beb14b79a195","Type":"ContainerStarted","Data":"2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c"}
Oct 30 22:51:20.970677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:20.970629    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss" podStartSLOduration=1.970616182 podStartE2EDuration="1.970616182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:20.919567289 +0000 UTC m=+4134.008305097" watchObservedRunningTime="2026-10-30 22:51:20.970616182 +0000 UTC m=+4134.059353984"
Oct 30 22:51:21.798365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:21.798334    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.815882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:21.815852    4410 generic.go:356] "Generic (PLEG): container finished" podID="85f6dfe9-57ad-428d-ac53-beb14b79a195" containerID="72b064a77c564f272739d783aabdbd77c735d3c9f6a8a7527c9adb6706d7c979" exitCode=0
Oct 30 22:51:21.816238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:21.815918    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss" event={"ID":"85f6dfe9-57ad-428d-ac53-beb14b79a195","Type":"ContainerDied","Data":"72b064a77c564f272739d783aabdbd77c735d3c9f6a8a7527c9adb6706d7c979"}
Oct 30 22:51:22.006599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:22.006562    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.230234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:22.230204    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.598647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:22.598601    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.961101 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:22.961068    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:23.044736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.044710    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\" (UniqueName: \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\") pod \"85f6dfe9-57ad-428d-ac53-beb14b79a195\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") "
Oct 30 22:51:23.044895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.044843    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\") pod \"85f6dfe9-57ad-428d-ac53-beb14b79a195\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") "
Oct 30 22:51:23.044945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.044899    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\") pod \"85f6dfe9-57ad-428d-ac53-beb14b79a195\" (UID: \"85f6dfe9-57ad-428d-ac53-beb14b79a195\") "
Oct 30 22:51:23.045242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.045218    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca" pod "85f6dfe9-57ad-428d-ac53-beb14b79a195" (UID: "85f6dfe9-57ad-428d-ac53-beb14b79a195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:23.046506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.046473    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:23.047039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.047016    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2" pod "85f6dfe9-57ad-428d-ac53-beb14b79a195" (UID: "85f6dfe9-57ad-428d-ac53-beb14b79a195"). InnerVolumeSpecName "kube-api-access-cwzk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:23.047140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.047062    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca" pod "85f6dfe9-57ad-428d-ac53-beb14b79a195" (UID: "85f6dfe9-57ad-428d-ac53-beb14b79a195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:23.145795 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.145746    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzk2\" (UniqueName: \"kubernetes.io/projected/85f6dfe9-57ad-428d-ac53-beb14b79a195-kube-api-access-cwzk2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.145954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.145797    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.145954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.145815    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f6dfe9-57ad-428d-ac53-beb14b79a195-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.311524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.311493    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"]
Oct 30 22:51:23.312000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.311983    4410 cpu_manager.go:414] "containerMap: removing container" podUID="85f6dfe9-57ad-428d-ac53-beb14b79a195" containerName="revoke-keys"
Oct 30 22:51:23.312000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.312000    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f6dfe9-57ad-428d-ac53-beb14b79a195" containerName="revoke-keys"
Oct 30 22:51:23.312145 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.312089    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f6dfe9-57ad-428d-ac53-beb14b79a195" containerName="revoke-keys"
Oct 30 22:51:23.315342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.315324    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.372560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.372533    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"]
Oct 30 22:51:23.449363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.449301    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hx8l\" (UniqueName: \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.449363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.449334    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.449532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.449426    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.550977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.550947    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hx8l\" (UniqueName: \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.551115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.550984    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.551115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.551028    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.551745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.551717    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.553571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.553545    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.609273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.609248    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hx8l\" (UniqueName: \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\") pod \"maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.628177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.628153    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:23.829089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.829008    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"
Oct 30 22:51:23.829252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.829012    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss" event={"ID":"85f6dfe9-57ad-428d-ac53-beb14b79a195","Type":"ContainerDied","Data":"2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c"}
Oct 30 22:51:23.829252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:23.829156    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dfd273cf442b63d014790932cf31d5678f86e1299223686a1f13206d9a8427c"
Oct 30 22:51:24.009557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.009156    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"]
Oct 30 22:51:24.437026 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.436994    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:24.630212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.630142    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:24.676373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.676339    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:24.816684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.816644    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:24.864204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.864161    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb" event={"ID":"d7e3e437-ecf8-4eca-a25b-ea402ca78749","Type":"ContainerStarted","Data":"6feabbbee2042dd06b7a8dd4b4b33bf0f202da7e11c3fb32e39b0487304d8b82"}
Oct 30 22:51:24.864432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.864231    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb" event={"ID":"d7e3e437-ecf8-4eca-a25b-ea402ca78749","Type":"ContainerStarted","Data":"f15ec1dcb8aa387e7674591f116545d6f87882fd99c21d4858e91fcf9730a390"}
Oct 30 22:51:24.993393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:24.993346    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb" podStartSLOduration=1.993332911 podStartE2EDuration="1.993332911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:24.99036415 +0000 UTC m=+4138.079101964" watchObservedRunningTime="2026-10-30 22:51:24.993332911 +0000 UTC m=+4138.082070714"
Oct 30 22:51:25.336415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.336332    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:25.493750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.493715    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:25.795750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.795722    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"]
Oct 30 22:51:25.799330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.799314    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.857038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.857004    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"]
Oct 30 22:51:25.872366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.872337    4410 generic.go:356] "Generic (PLEG): container finished" podID="d7e3e437-ecf8-4eca-a25b-ea402ca78749" containerID="6feabbbee2042dd06b7a8dd4b4b33bf0f202da7e11c3fb32e39b0487304d8b82" exitCode=0
Oct 30 22:51:25.872505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.872421    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb" event={"ID":"d7e3e437-ecf8-4eca-a25b-ea402ca78749","Type":"ContainerDied","Data":"6feabbbee2042dd06b7a8dd4b4b33bf0f202da7e11c3fb32e39b0487304d8b82"}
Oct 30 22:51:25.873838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.873819    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.873939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.873925    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cb7\" (UniqueName: \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.874046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.874029    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.975576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.975542    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cb7\" (UniqueName: \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.975796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.975619    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.975796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.975689    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.976320 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.976294    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:25.978085 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:25.978067    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:26.032913 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.032865    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cb7\" (UniqueName: \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\") pod \"maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:26.112531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.112456    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:26.486252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.486222    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"]
Oct 30 22:51:26.487873 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:51:26.487845    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bde9c9_8029_4f63_8ff5_e9c0b4b551c5.slice/crio-6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9 WatchSource:0}: Error finding container 6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9: Status 404 returned error can't find the container with id 6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9
Oct 30 22:51:26.880789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.880739    4410 generic.go:356] "Generic (PLEG): container finished" podID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerID="4312a5fd83b5a495b18305edf15f2066de2d4668b118c3b890696385d7bbb0a7" exitCode=0
Oct 30 22:51:26.880997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.880812    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" event={"ID":"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf","Type":"ContainerDied","Data":"4312a5fd83b5a495b18305edf15f2066de2d4668b118c3b890696385d7bbb0a7"}
Oct 30 22:51:26.882658 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.882631    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45" event={"ID":"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5","Type":"ContainerStarted","Data":"190bbbfc7fffb361a69a2cf1f54634cd42f0fed12bc935635396be0461244b7b"}
Oct 30 22:51:26.882824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.882680    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45" event={"ID":"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5","Type":"ContainerStarted","Data":"6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9"}
Oct 30 22:51:26.896599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.896579    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:51:26.985637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.985609    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\") pod \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") "
Oct 30 22:51:26.985844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.985656    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\" (UniqueName: \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\") pod \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\" (UID: \"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf\") "
Oct 30 22:51:26.986108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.986069    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume" pod "8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" (UID: "8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:26.986399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.986363    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:26.988930 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:26.988896    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg" pod "8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" (UID: "8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf"). InnerVolumeSpecName "kube-api-access-vmctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:27.000087 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.000043    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45" podStartSLOduration=2.000029742 podStartE2EDuration="2.000029742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:26.998094676 +0000 UTC m=+4140.086832482" watchObservedRunningTime="2026-10-30 22:51:27.000029742 +0000 UTC m=+4140.088767565"
Oct 30 22:51:27.030863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.030840    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:27.087481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087457    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\") pod \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") "
Oct 30 22:51:27.087622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087496    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\" (UniqueName: \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\") pod \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") "
Oct 30 22:51:27.087622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087551    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\") pod \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\" (UID: \"d7e3e437-ecf8-4eca-a25b-ea402ca78749\") "
Oct 30 22:51:27.087844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087821    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca" pod "d7e3e437-ecf8-4eca-a25b-ea402ca78749" (UID: "d7e3e437-ecf8-4eca-a25b-ea402ca78749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:27.087972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087950    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmctg\" (UniqueName: \"kubernetes.io/projected/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf-kube-api-access-vmctg\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.087972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.087973    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.089409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.089390    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l" pod "d7e3e437-ecf8-4eca-a25b-ea402ca78749" (UID: "d7e3e437-ecf8-4eca-a25b-ea402ca78749"). InnerVolumeSpecName "kube-api-access-9hx8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:27.089599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.089579    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca" pod "d7e3e437-ecf8-4eca-a25b-ea402ca78749" (UID: "d7e3e437-ecf8-4eca-a25b-ea402ca78749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:27.188833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.188804    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e3e437-ecf8-4eca-a25b-ea402ca78749-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.188833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.188830    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hx8l\" (UniqueName: \"kubernetes.io/projected/d7e3e437-ecf8-4eca-a25b-ea402ca78749-kube-api-access-9hx8l\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.552407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.552375    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 22:51:27.642297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.642268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:27.889990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.889961    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"
Oct 30 22:51:27.890178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.889965    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb" event={"ID":"d7e3e437-ecf8-4eca-a25b-ea402ca78749","Type":"ContainerDied","Data":"f15ec1dcb8aa387e7674591f116545d6f87882fd99c21d4858e91fcf9730a390"}
Oct 30 22:51:27.890178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.890090    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15ec1dcb8aa387e7674591f116545d6f87882fd99c21d4858e91fcf9730a390"
Oct 30 22:51:27.891247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.891229    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"
Oct 30 22:51:27.891376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.891273    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx" event={"ID":"8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf","Type":"ContainerDied","Data":"705c055be9ca260d72539ac3050180e65639214186818ce9ca3d460bfd6546fc"}
Oct 30 22:51:27.891376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.891322    4410 scope.go:122] "RemoveContainer" containerID="4312a5fd83b5a495b18305edf15f2066de2d4668b118c3b890696385d7bbb0a7"
Oct 30 22:51:27.893044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.893017    4410 generic.go:356] "Generic (PLEG): container finished" podID="54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" containerID="190bbbfc7fffb361a69a2cf1f54634cd42f0fed12bc935635396be0461244b7b" exitCode=0
Oct 30 22:51:27.893160 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:27.893073    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45" event={"ID":"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5","Type":"ContainerDied","Data":"190bbbfc7fffb361a69a2cf1f54634cd42f0fed12bc935635396be0461244b7b"}
Oct 30 22:51:28.053535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:28.053508    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:51:28.078054 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:28.078028    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a0116e-76544d48bc-5czsx"]
Oct 30 22:51:28.384665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:28.384581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:29.020736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.020700    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" path="/var/lib/kubelet/pods/8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf/volumes"
Oct 30 22:51:29.042643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.042620    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:29.108409 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.108382    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\") pod \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") "
Oct 30 22:51:29.108543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.108434    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\" (UniqueName: \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\") pod \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") "
Oct 30 22:51:29.108593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.108547    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\") pod \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\" (UID: \"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5\") "
Oct 30 22:51:29.108820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.108765    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca" pod "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" (UID: "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:29.109118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.109093    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:29.110486 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.110463    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7" pod "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" (UID: "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5"). InnerVolumeSpecName "kube-api-access-g4cb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:29.110585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.110522    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca" pod "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" (UID: "54bde9c9-8029-4f63-8ff5-e9c0b4b551c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:29.209727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.209667    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cb7\" (UniqueName: \"kubernetes.io/projected/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-kube-api-access-g4cb7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:29.209727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.209692    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:29.776634 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.776604    4410 scope.go:122] "RemoveContainer" containerID="887aabc00bfc95ff42a4196b3a9cae167be8d98702d1ab0c463bb302179ee53e"
Oct 30 22:51:29.786610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.786588    4410 scope.go:122] "RemoveContainer" containerID="ed5f4d849273212c55d9a4511543d46de9ab28141d553c0a4996a752b8fdf1c0"
Oct 30 22:51:29.905485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.905457    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"
Oct 30 22:51:29.905485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.905468    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45" event={"ID":"54bde9c9-8029-4f63-8ff5-e9c0b4b551c5","Type":"ContainerDied","Data":"6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9"}
Oct 30 22:51:29.905685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:29.905517    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6563f2ec769310dd209254877ff45b0a0622237e5e9bcb58963be843ecd998c9"
Oct 30 22:51:30.136568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.136487    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:30.283335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.283301    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:51:30.287929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.287904    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:51:30.332668 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.332639    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 22:51:30.334781 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.334746    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 22:51:30.400928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:30.400900    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:31.510945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:31.510911    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:31.720254 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:31.720217    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:33.306561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.306501    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:51:33.458014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.457985    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\" (UniqueName: \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\") pod \"694538c8-6f7b-4081-8908-59124e60c1cc\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") "
Oct 30 22:51:33.458218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.458164    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\") pod \"694538c8-6f7b-4081-8908-59124e60c1cc\" (UID: \"694538c8-6f7b-4081-8908-59124e60c1cc\") "
Oct 30 22:51:33.458482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.458459    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume" pod "694538c8-6f7b-4081-8908-59124e60c1cc" (UID: "694538c8-6f7b-4081-8908-59124e60c1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:33.460137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.460114    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7" pod "694538c8-6f7b-4081-8908-59124e60c1cc" (UID: "694538c8-6f7b-4081-8908-59124e60c1cc"). InnerVolumeSpecName "kube-api-access-84sx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:33.559280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.559216    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694538c8-6f7b-4081-8908-59124e60c1cc-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:33.559280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.559253    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sx7\" (UniqueName: \"kubernetes.io/projected/694538c8-6f7b-4081-8908-59124e60c1cc-kube-api-access-84sx7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:33.751383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.751347    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:33.930770 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.930729    4410 generic.go:356] "Generic (PLEG): container finished" podID="694538c8-6f7b-4081-8908-59124e60c1cc" containerID="f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6" exitCode=0
Oct 30 22:51:33.930770 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.930792    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"
Oct 30 22:51:33.931033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.930807    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" event={"ID":"694538c8-6f7b-4081-8908-59124e60c1cc","Type":"ContainerDied","Data":"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"}
Oct 30 22:51:33.931033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.930861    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87" event={"ID":"694538c8-6f7b-4081-8908-59124e60c1cc","Type":"ContainerDied","Data":"21355174fc807dc4aacb10d7b56aab84f4cd3771123ce4325281fe1b404b03a0"}
Oct 30 22:51:33.931033 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.930885    4410 scope.go:122] "RemoveContainer" containerID="f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"
Oct 30 22:51:33.940706 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.940687    4410 scope.go:122] "RemoveContainer" containerID="f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"
Oct 30 22:51:33.941059 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:33.941041    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6\": container with ID starting with f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6 not found: ID does not exist" containerID="f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"
Oct 30 22:51:33.941143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:33.941074    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6"} err="failed to get container status \"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6\": rpc error: code = NotFound desc = could not find container \"f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6\": container with ID starting with f4222d6fee425b0ff9711ac1c4e4af59e06612d89b43b72506a74dfd02f929b6 not found: ID does not exist"
Oct 30 22:51:34.096105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:34.096075    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:51:34.144695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:34.144665    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9f7654-694985699f-f7g87"]
Oct 30 22:51:35.018307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:35.018277    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" path="/var/lib/kubelet/pods/694538c8-6f7b-4081-8908-59124e60c1cc/volumes"
Oct 30 22:51:35.256325 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:35.256294    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:35.419115 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:35.419083    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:37.264510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:37.264474    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:37.714579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:37.714539    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:37.979432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:37.979345    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:38.656327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:38.656291    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:42.572528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.572480    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main" containerID="cri-o://8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3" gracePeriod=45
Oct 30 22:51:42.838594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.838524    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:51:42.958812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.958297    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.958872    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.958998    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\" (UniqueName: \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.959078    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.959132    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959257 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.959165    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\") pod \"8f734874-6279-4e14-962d-10fe1c71ac16\" (UID: \"8f734874-6279-4e14-962d-10fe1c71ac16\") "
Oct 30 22:51:42.959755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.959407    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:42.959755 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.959636    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:42.961492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.961453    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:42.961731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.961709    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:42.962198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.962168    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "kube-api-access-h6rvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:42.963238 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.963213    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f734874-6279-4e14-962d-10fe1c71ac16-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.963335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.963242    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.963335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.963257    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rvl\" (UniqueName: \"kubernetes.io/projected/8f734874-6279-4e14-962d-10fe1c71ac16-kube-api-access-h6rvl\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.963335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.963279    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.963335 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.963293    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.967864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.966190    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm" pod "8f734874-6279-4e14-962d-10fe1c71ac16" (UID: "8f734874-6279-4e14-962d-10fe1c71ac16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:42.986243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.986208    4410 generic.go:356] "Generic (PLEG): container finished" podID="8f734874-6279-4e14-962d-10fe1c71ac16" containerID="8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3" exitCode=0
Oct 30 22:51:42.986382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.986275    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"
Oct 30 22:51:42.986382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.986294    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" event={"ID":"8f734874-6279-4e14-962d-10fe1c71ac16","Type":"ContainerDied","Data":"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"}
Oct 30 22:51:42.986382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.986345    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd" event={"ID":"8f734874-6279-4e14-962d-10fe1c71ac16","Type":"ContainerDied","Data":"16cfcbbd9db61dce8a82b74aa2112e8243bc94ad40bbb5d010919d38b582d533"}
Oct 30 22:51:42.986382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:42.986368    4410 scope.go:122] "RemoveContainer" containerID="8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"
Oct 30 22:51:43.001945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.001919    4410 scope.go:122] "RemoveContainer" containerID="8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"
Oct 30 22:51:43.002265 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:43.002236    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3\": container with ID starting with 8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3 not found: ID does not exist" containerID="8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"
Oct 30 22:51:43.002367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.002279    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3"} err="failed to get container status \"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3\": rpc error: code = NotFound desc = could not find container \"8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3\": container with ID starting with 8f463467bc901a9823cf6354f7ae811634106c249673b3ddcf9940585ffa50c3 not found: ID does not exist"
Oct 30 22:51:43.064565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.064537    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f734874-6279-4e14-962d-10fe1c71ac16-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.148699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.148670    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 22:51:43.152245 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.152221    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/premium-w1-1c3eef-kserve-d586cf5f6-fn2dd"]
Oct 30 22:51:43.540473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:43.540391    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:45.018732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.018694    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" path="/var/lib/kubelet/pods/8f734874-6279-4e14-962d-10fe1c71ac16/volumes"
Oct 30 22:51:45.306273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.306196    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main" containerID="cri-o://57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04" gracePeriod=45
Oct 30 22:51:45.307327 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.307299    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main" containerID="cri-o://ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f" gracePeriod=45
Oct 30 22:51:45.351769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.351739    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main" containerID="cri-o://1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201" gracePeriod=45
Oct 30 22:51:45.353338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.353315    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main" containerID="cri-o://01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c" gracePeriod=45
Oct 30 22:51:45.639805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.633614    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:51:45.648334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.648313    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:51:45.655837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.655814    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:51:45.659543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.659517    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:51:45.792528 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792491    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.792722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792550    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.792722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792591    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\" (UniqueName: \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.792722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792620    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.792722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792671    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.792722 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792699    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792727    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792757    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792816    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792836    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792842    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792867    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792904    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792938    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.793014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.792985    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793029    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\" (UniqueName: \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793079    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793088    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793125    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793155    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\") pod \"4eef84fe-4003-47f8-b058-f3f6e58369f0\" (UID: \"4eef84fe-4003-47f8-b058-f3f6e58369f0\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793203    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793265    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\" (UniqueName: \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793329    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793388    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\" (UniqueName: \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793422    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\") pod \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\" (UID: \"2e469d15-9f4a-4ac3-a422-b8509c7971f8\") "
Oct 30 22:51:45.793519 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793495    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\") pod \"5e4c12a7-6a75-4c34-ad98-342b571977a0\" (UID: \"5e4c12a7-6a75-4c34-ad98-342b571977a0\") "
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793548    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\") pod \"e36673df-c797-4dba-866d-8351c430c91f\" (UID: \"e36673df-c797-4dba-866d-8351c430c91f\") "
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793585    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793752    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.793957    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794206    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794228    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794233    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794242    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794256    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794270    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794269    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794283    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794419    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.794679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794485    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.795472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794900    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.795472 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.794921    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.796248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.796198    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:45.796621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.796594    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.797403 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.797354    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.798416 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.798378    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "kube-api-access-cgnkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:45.798569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.798541    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:45.799003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.798968    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:45.799104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799061    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2" pod "5e4c12a7-6a75-4c34-ad98-342b571977a0" (UID: "5e4c12a7-6a75-4c34-ad98-342b571977a0"). InnerVolumeSpecName "kube-api-access-gxpg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:45.799104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799058    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.799225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799183    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm" pod "2e469d15-9f4a-4ac3-a422-b8509c7971f8" (UID: "2e469d15-9f4a-4ac3-a422-b8509c7971f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:45.799288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799272    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:45.799342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799275    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd" pod "e36673df-c797-4dba-866d-8351c430c91f" (UID: "e36673df-c797-4dba-866d-8351c430c91f"). InnerVolumeSpecName "kube-api-access-gxshd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:45.799522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.799495    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj" pod "4eef84fe-4003-47f8-b058-f3f6e58369f0" (UID: "4eef84fe-4003-47f8-b058-f3f6e58369f0"). InnerVolumeSpecName "kube-api-access-452pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:45.895550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895518    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452pj\" (UniqueName: \"kubernetes.io/projected/4eef84fe-4003-47f8-b058-f3f6e58369f0-kube-api-access-452pj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895551    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895567    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36673df-c797-4dba-866d-8351c430c91f-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895578    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895590    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4c12a7-6a75-4c34-ad98-342b571977a0-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895602    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eef84fe-4003-47f8-b058-f3f6e58369f0-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895613    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36673df-c797-4dba-866d-8351c430c91f-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895630    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895643    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxpg2\" (UniqueName: \"kubernetes.io/projected/5e4c12a7-6a75-4c34-ad98-342b571977a0-kube-api-access-gxpg2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895655    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895667    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895678    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eef84fe-4003-47f8-b058-f3f6e58369f0-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895689    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895704    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgnkb\" (UniqueName: \"kubernetes.io/projected/2e469d15-9f4a-4ac3-a422-b8509c7971f8-kube-api-access-cgnkb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895715    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895727    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxshd\" (UniqueName: \"kubernetes.io/projected/e36673df-c797-4dba-866d-8351c430c91f-kube-api-access-gxshd\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895739    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e469d15-9f4a-4ac3-a422-b8509c7971f8-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.895818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:45.895754    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4c12a7-6a75-4c34-ad98-342b571977a0-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:46.007914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.007881    4410 generic.go:356] "Generic (PLEG): container finished" podID="e36673df-c797-4dba-866d-8351c430c91f" containerID="01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c" exitCode=0
Oct 30 22:51:46.008109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.007919    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"
Oct 30 22:51:46.008109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.007967    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" event={"ID":"e36673df-c797-4dba-866d-8351c430c91f","Type":"ContainerDied","Data":"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"}
Oct 30 22:51:46.008109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.008029    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz" event={"ID":"e36673df-c797-4dba-866d-8351c430c91f","Type":"ContainerDied","Data":"fab8da2af6aa7318ad07771a6bcd92a553969ed1e0ce31bdc35d063c0653ad28"}
Oct 30 22:51:46.008109 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.008055    4410 scope.go:122] "RemoveContainer" containerID="01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"
Oct 30 22:51:46.009693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.009668    4410 generic.go:356] "Generic (PLEG): container finished" podID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerID="ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f" exitCode=0
Oct 30 22:51:46.009845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.009769    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" event={"ID":"2e469d15-9f4a-4ac3-a422-b8509c7971f8","Type":"ContainerDied","Data":"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"}
Oct 30 22:51:46.009845 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.009833    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd" event={"ID":"2e469d15-9f4a-4ac3-a422-b8509c7971f8","Type":"ContainerDied","Data":"00150b309f267aa6eb7a6927a72db4b97217db766d966bd7e073878ffc74eb8b"}
Oct 30 22:51:46.009985 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.009863    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"
Oct 30 22:51:46.013996 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.013972    4410 generic.go:356] "Generic (PLEG): container finished" podID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerID="57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04" exitCode=0
Oct 30 22:51:46.014341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.014054    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" event={"ID":"5e4c12a7-6a75-4c34-ad98-342b571977a0","Type":"ContainerDied","Data":"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"}
Oct 30 22:51:46.014341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.014083    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7" event={"ID":"5e4c12a7-6a75-4c34-ad98-342b571977a0","Type":"ContainerDied","Data":"588400eb8cf830512613667ecadac28fa6f033f1eb0a7802454c8cec42a945ef"}
Oct 30 22:51:46.014341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.014142    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"
Oct 30 22:51:46.016122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.016101    4410 generic.go:356] "Generic (PLEG): container finished" podID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerID="1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201" exitCode=0
Oct 30 22:51:46.016234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.016142    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"
Oct 30 22:51:46.016234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.016174    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" event={"ID":"4eef84fe-4003-47f8-b058-f3f6e58369f0","Type":"ContainerDied","Data":"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"}
Oct 30 22:51:46.016234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.016201    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48" event={"ID":"4eef84fe-4003-47f8-b058-f3f6e58369f0","Type":"ContainerDied","Data":"e2393eab6adcfd53b1307bf737401084871ac547e4e96355fb57343540420019"}
Oct 30 22:51:46.026577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.026546    4410 scope.go:122] "RemoveContainer" containerID="01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"
Oct 30 22:51:46.026930 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:46.026907    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c\": container with ID starting with 01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c not found: ID does not exist" containerID="01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"
Oct 30 22:51:46.026999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.026944    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c"} err="failed to get container status \"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c\": rpc error: code = NotFound desc = could not find container \"01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c\": container with ID starting with 01f47a22db74e49b88c29eda38ffe28275bd93e2fb453acd909383f5f6ea093c not found: ID does not exist"
Oct 30 22:51:46.026999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.026967    4410 scope.go:122] "RemoveContainer" containerID="ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"
Oct 30 22:51:46.038377 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.038346    4410 scope.go:122] "RemoveContainer" containerID="ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"
Oct 30 22:51:46.038626 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:46.038608    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f\": container with ID starting with ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f not found: ID does not exist" containerID="ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"
Oct 30 22:51:46.038682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.038638    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f"} err="failed to get container status \"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f\": rpc error: code = NotFound desc = could not find container \"ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f\": container with ID starting with ddc54901952038c9b0d4be9dbdd50f55f51767816b28b877d1493e77d31d951f not found: ID does not exist"
Oct 30 22:51:46.038682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.038660    4410 scope.go:122] "RemoveContainer" containerID="57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"
Oct 30 22:51:46.047146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.047130    4410 scope.go:122] "RemoveContainer" containerID="57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"
Oct 30 22:51:46.047423 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:46.047394    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04\": container with ID starting with 57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04 not found: ID does not exist" containerID="57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"
Oct 30 22:51:46.047496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.047423    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04"} err="failed to get container status \"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04\": rpc error: code = NotFound desc = could not find container \"57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04\": container with ID starting with 57b4aa1564b1b1457115bb425202cdf1c5450df06cc3a20c917612ec424a7c04 not found: ID does not exist"
Oct 30 22:51:46.047496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.047440    4410 scope.go:122] "RemoveContainer" containerID="1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"
Oct 30 22:51:46.058351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.058329    4410 scope.go:122] "RemoveContainer" containerID="1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"
Oct 30 22:51:46.058870 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:46.058851    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201\": container with ID starting with 1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201 not found: ID does not exist" containerID="1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"
Oct 30 22:51:46.058945 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.058877    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201"} err="failed to get container status \"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201\": rpc error: code = NotFound desc = could not find container \"1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201\": container with ID starting with 1b42386b7c1169a2e2a0033a5e10a507bd9c73f3716646b8c6708a455f448201 not found: ID does not exist"
Oct 30 22:51:46.196850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.196740    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 22:51:46.206764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.206730    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/premium-w4-d2a3e0-kserve-7d76b67b47-9gq48"]
Oct 30 22:51:46.310840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.310767    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:51:46.315463 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.315433    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-w4-d2a3e0-kserve-5589ff79bb-gx8wd"]
Oct 30 22:51:46.421840 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.421804    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:51:46.427466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.427440    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/distinct-2-w4-d2a3e0-kserve-688f6866f-sljc7"]
Oct 30 22:51:46.534681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.534612    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 22:51:46.538316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:46.538294    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d2a3e0/simulator-w4-d2a3e0-kserve-88b9d7f94-nc9jz"]
Oct 30 22:51:47.019288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.019254    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" path="/var/lib/kubelet/pods/2e469d15-9f4a-4ac3-a422-b8509c7971f8/volumes"
Oct 30 22:51:47.019704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.019684    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" path="/var/lib/kubelet/pods/4eef84fe-4003-47f8-b058-f3f6e58369f0/volumes"
Oct 30 22:51:47.020133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.020118    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" path="/var/lib/kubelet/pods/5e4c12a7-6a75-4c34-ad98-342b571977a0/volumes"
Oct 30 22:51:47.020751 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.020737    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36673df-c797-4dba-866d-8351c430c91f" path="/var/lib/kubelet/pods/e36673df-c797-4dba-866d-8351c430c91f/volumes"
Oct 30 22:51:47.701842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.701807    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"]
Oct 30 22:51:47.702228 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702217    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing"
Oct 30 22:51:47.702272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702229    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing"
Oct 30 22:51:47.702272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702242    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main"
Oct 30 22:51:47.702272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702247    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main"
Oct 30 22:51:47.702272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702262    4410 cpu_manager.go:414] "containerMap: removing container" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing"
Oct 30 22:51:47.702272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702268    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702279    4410 cpu_manager.go:414] "containerMap: removing container" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702283    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702292    4410 cpu_manager.go:414] "containerMap: removing container" podUID="54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" containerName="revoke-keys"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702297    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" containerName="revoke-keys"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702309    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702313    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702320    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702324    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702337    4410 cpu_manager.go:414] "containerMap: removing container" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702341    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702348    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d7e3e437-ecf8-4eca-a25b-ea402ca78749" containerName="revoke-keys"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702353    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e3e437-ecf8-4eca-a25b-ea402ca78749" containerName="revoke-keys"
Oct 30 22:51:47.702433 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702429    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" containerName="revoke-keys"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702439    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36673df-c797-4dba-866d-8351c430c91f" containerName="main"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702446    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f734874-6279-4e14-962d-10fe1c71ac16" containerName="main"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702453    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eef84fe-4003-47f8-b058-f3f6e58369f0" containerName="main"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702459    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e3e437-ecf8-4eca-a25b-ea402ca78749" containerName="revoke-keys"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702465    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d111bba-f68c-4ca3-9fc8-21c3d5b80eaf" containerName="payload-pre-processing"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702471    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e469d15-9f4a-4ac3-a422-b8509c7971f8" containerName="main"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702477    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="694538c8-6f7b-4081-8908-59124e60c1cc" containerName="payload-processing"
Oct 30 22:51:47.702856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.702483    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4c12a7-6a75-4c34-ad98-342b571977a0" containerName="main"
Oct 30 22:51:47.707361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.707344    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.759298 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.759270    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"]
Oct 30 22:51:47.763270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.763243    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"]
Oct 30 22:51:47.763425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.763372    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.806444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.806417    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:47.806444 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.806447    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:51:47.809006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.808982    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:47.809006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.808988    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:47.817936 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.817908    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrjh\" (UniqueName: \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.818072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.818058    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skngw\" (UniqueName: \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.818140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.818109    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.818214 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.818197    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.818264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.818236    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.818315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.818268    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.820078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.820057    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"]
Oct 30 22:51:47.919060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919034    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.919204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919068    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.919204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919092    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.919332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919208    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrjh\" (UniqueName: \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.919332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919311    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skngw\" (UniqueName: \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.919447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919387    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.919730 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919707    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.919842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.919784    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.921446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.921427    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:47.921662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.921643    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.976387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.976319    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skngw\" (UniqueName: \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\") pod \"maas-api-revoke-sub-f066e2cf-x457k\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:47.976489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:47.976469    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrjh\" (UniqueName: \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\") pod \"maas-api-revoke-sub-a6d90e87-stp6b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:48.020288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:48.020263    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:48.078736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:48.078709    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:48.392192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:48.392168    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"]
Oct 30 22:51:48.393720 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:51:48.393696    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a91940_6108_4e72_8bfa_3c1ba04f4d5b.slice/crio-211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484 WatchSource:0}: Error finding container 211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484: Status 404 returned error can't find the container with id 211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484
Oct 30 22:51:48.493435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:48.493347    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"]
Oct 30 22:51:48.495982 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:51:48.495958    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293d7c07_547f_4636_9eb4_da2565a6df36.slice/crio-4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf WatchSource:0}: Error finding container 4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf: Status 404 returned error can't find the container with id 4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf
Oct 30 22:51:48.563162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:48.563135    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:49.037296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.037212    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b" event={"ID":"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b","Type":"ContainerStarted","Data":"30f7d11ea51cd147486045a176a40a1559d8951d8533cde686a3e7a0f232d4c8"}
Oct 30 22:51:49.037296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.037260    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b" event={"ID":"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b","Type":"ContainerStarted","Data":"211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484"}
Oct 30 22:51:49.038922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.038897    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k" event={"ID":"293d7c07-547f-4636-9eb4-da2565a6df36","Type":"ContainerStarted","Data":"f2a7b85010380a0e47d14753d426e8c2605a71f63523b5800d3b746444996ddf"}
Oct 30 22:51:49.039034 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.038934    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k" event={"ID":"293d7c07-547f-4636-9eb4-da2565a6df36","Type":"ContainerStarted","Data":"4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf"}
Oct 30 22:51:49.155088 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.155041    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b" podStartSLOduration=2.155027315 podStartE2EDuration="2.155027315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:49.152468416 +0000 UTC m=+4162.241206219" watchObservedRunningTime="2026-10-30 22:51:49.155027315 +0000 UTC m=+4162.243765117"
Oct 30 22:51:49.267480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:49.267431    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k" podStartSLOduration=2.267418486 podStartE2EDuration="2.267418486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:49.265495724 +0000 UTC m=+4162.354233551" watchObservedRunningTime="2026-10-30 22:51:49.267418486 +0000 UTC m=+4162.356156322"
Oct 30 22:51:50.046442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:50.046400    4410 generic.go:356] "Generic (PLEG): container finished" podID="a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" containerID="30f7d11ea51cd147486045a176a40a1559d8951d8533cde686a3e7a0f232d4c8" exitCode=0
Oct 30 22:51:50.046896 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:50.046454    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b" event={"ID":"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b","Type":"ContainerDied","Data":"30f7d11ea51cd147486045a176a40a1559d8951d8533cde686a3e7a0f232d4c8"}
Oct 30 22:51:50.048111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:50.048088    4410 generic.go:356] "Generic (PLEG): container finished" podID="293d7c07-547f-4636-9eb4-da2565a6df36" containerID="f2a7b85010380a0e47d14753d426e8c2605a71f63523b5800d3b746444996ddf" exitCode=0
Oct 30 22:51:50.048284 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:50.048119    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k" event={"ID":"293d7c07-547f-4636-9eb4-da2565a6df36","Type":"ContainerDied","Data":"f2a7b85010380a0e47d14753d426e8c2605a71f63523b5800d3b746444996ddf"}
Oct 30 22:51:50.521552 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:51:50.521516    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:51:51.209916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.209885    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:51.212786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.212751    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:51.252328 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252302    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\") pod \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") "
Oct 30 22:51:51.252445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252359    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\") pod \"293d7c07-547f-4636-9eb4-da2565a6df36\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") "
Oct 30 22:51:51.252445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252404    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\") pod \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") "
Oct 30 22:51:51.252538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252487    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\" (UniqueName: \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\") pod \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\" (UID: \"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b\") "
Oct 30 22:51:51.252577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252535    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\" (UniqueName: \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\") pod \"293d7c07-547f-4636-9eb4-da2565a6df36\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") "
Oct 30 22:51:51.252577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252568    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\") pod \"293d7c07-547f-4636-9eb4-da2565a6df36\" (UID: \"293d7c07-547f-4636-9eb4-da2565a6df36\") "
Oct 30 22:51:51.252794 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252745    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca" pod "293d7c07-547f-4636-9eb4-da2565a6df36" (UID: "293d7c07-547f-4636-9eb4-da2565a6df36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:51.252914 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.252828    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca" pod "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" (UID: "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:51.253117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.253099    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:51.253195 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.253124    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:51.254643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.254622    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca" pod "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" (UID: "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:51.255069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.255047    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw" pod "293d7c07-547f-4636-9eb4-da2565a6df36" (UID: "293d7c07-547f-4636-9eb4-da2565a6df36"). InnerVolumeSpecName "kube-api-access-skngw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:51.255156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.255086    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca" pod "293d7c07-547f-4636-9eb4-da2565a6df36" (UID: "293d7c07-547f-4636-9eb4-da2565a6df36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:51.255156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.255131    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh" pod "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" (UID: "a6a91940-6108-4e72-8bfa-3c1ba04f4d5b"). InnerVolumeSpecName "kube-api-access-qgrjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:51.354341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.354275    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrjh\" (UniqueName: \"kubernetes.io/projected/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-kube-api-access-qgrjh\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:51.354341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.354313    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skngw\" (UniqueName: \"kubernetes.io/projected/293d7c07-547f-4636-9eb4-da2565a6df36-kube-api-access-skngw\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:51.354341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.354331    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293d7c07-547f-4636-9eb4-da2565a6df36-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:51.354535 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:51.354347    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:52.062384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.062357    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"
Oct 30 22:51:52.062384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.062372    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k" event={"ID":"293d7c07-547f-4636-9eb4-da2565a6df36","Type":"ContainerDied","Data":"4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf"}
Oct 30 22:51:52.062693 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.062416    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f39dd7733aa9fbb870458131573f24595a5aa07216098a45bcbcbc8173b4acf"
Oct 30 22:51:52.063888 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.063861    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b" event={"ID":"a6a91940-6108-4e72-8bfa-3c1ba04f4d5b","Type":"ContainerDied","Data":"211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484"}
Oct 30 22:51:52.064002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.063890    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"
Oct 30 22:51:52.064002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:52.063897    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211261535b457f792a24a6d63cb5fa6044825f476df9865235c6563e53bf6484"
Oct 30 22:51:53.452006 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:53.451966    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:58.296485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:51:58.296446    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:02.492266 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:02.492225    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:03.663122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:03.663075    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:04.013363 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.013332    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:04.139196 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.139164    4410 generic.go:356] "Generic (PLEG): container finished" podID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerID="468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532" exitCode=0
Oct 30 22:52:04.139371 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.139240    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerDied","Data":"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"}
Oct 30 22:52:04.139681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.139652    4410 scope.go:122] "RemoveContainer" containerID="468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"
Oct 30 22:52:04.142061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.142029    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:52:04.142191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.142112    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:52:04.142191 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.142163    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:52:04.274268 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.274182    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:04.488881 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.488845    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"]
Oct 30 22:52:04.491389 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.491366    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a0116e-xcrxb"]
Oct 30 22:52:04.868647 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:04.868595    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.018352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.018318    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5567c83-5236-4c5b-8b4e-83122149c936" path="/var/lib/kubelet/pods/b5567c83-5236-4c5b-8b4e-83122149c936/volumes"
Oct 30 22:52:05.181911 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.181872    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerStarted","Data":"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"}
Oct 30 22:52:05.206629 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.206599    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.276801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.257082    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:52:05.276801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.257192    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:52:05.276801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.257233    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:52:05.370915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.370881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.530351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.530268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.686809 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.686754    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.824598 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.824515    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"]
Oct 30 22:52:05.826731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.826704    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e90906-nmwss"]
Oct 30 22:52:05.867664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:05.867639    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:06.175137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:06.175105    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:06.264150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:06.264117    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:52:06.264419 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:06.264388    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:52:06.582154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:06.579722    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:06.742923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:06.742888    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:07.018959 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:07.018926    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f6dfe9-57ad-428d-ac53-beb14b79a195" path="/var/lib/kubelet/pods/85f6dfe9-57ad-428d-ac53-beb14b79a195/volumes"
Oct 30 22:52:07.268931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:07.268886    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:52:07.269589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:07.269509    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:52:08.263798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:08.259139    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:52:08.263798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:08.259216    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:52:09.255175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:09.255139    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:52:09.255764 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:09.255731    4410 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:52:09.687456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:09.687417    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.299575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:10.299526    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:52:10.300144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:10.299945    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:52:10.300615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:10.300591    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:52:11.057090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:11.057056    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.243307 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:11.243254    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.416434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:11.416403    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:12.042107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.042077    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:12.231200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231168    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"]
Oct 30 22:52:12.231575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231561    4410 cpu_manager.go:414] "containerMap: removing container" podUID="293d7c07-547f-4636-9eb4-da2565a6df36" containerName="revoke-keys"
Oct 30 22:52:12.231622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231576    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293d7c07-547f-4636-9eb4-da2565a6df36" containerName="revoke-keys"
Oct 30 22:52:12.231622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231590    4410 cpu_manager.go:414] "containerMap: removing container" podUID="a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" containerName="revoke-keys"
Oct 30 22:52:12.231622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231595    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" containerName="revoke-keys"
Oct 30 22:52:12.231723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231691    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="293d7c07-547f-4636-9eb4-da2565a6df36" containerName="revoke-keys"
Oct 30 22:52:12.231723 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.231700    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" containerName="revoke-keys"
Oct 30 22:52:12.234895 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.234876    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.286731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.286709    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:12.336170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.336085    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:52:12.336392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.336128    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:12.336392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.336176    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:12.339922 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.339896    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"]
Oct 30 22:52:12.417728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.417697    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.418138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.417739    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8gx\" (UniqueName: \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.418138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.417840    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.519441 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.519408    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.519614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.519529    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.519614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.519553    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8gx\" (UniqueName: \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.520095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.520071    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.521779 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.521750    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.575662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.575633    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8gx\" (UniqueName: \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:12.848131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:12.848100    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:13.140111 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.140075    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.217063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.217040    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"]
Oct 30 22:52:13.219339 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:52:13.219309    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5798df_3433_4aa9_8801_58f0d34e340f.slice/crio-03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5 WatchSource:0}: Error finding container 03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5: Status 404 returned error can't find the container with id 03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5
Oct 30 22:52:13.230480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.230451    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv" event={"ID":"fd5798df-3433-4aa9-8801-58f0d34e340f","Type":"ContainerStarted","Data":"03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5"}
Oct 30 22:52:13.303383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.303355    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.465907 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.465881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.623440 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.623407    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.796510 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:13.796420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:14.237728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.237694    4410 generic.go:356] "Generic (PLEG): container finished" podID="fd5798df-3433-4aa9-8801-58f0d34e340f" containerID="4ce5eb7249d07f9d5a8bae5c243853926fec914f70cc866310d8cb349ee22ed1" exitCode=0
Oct 30 22:52:14.237887 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.237746    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv" event={"ID":"fd5798df-3433-4aa9-8801-58f0d34e340f","Type":"ContainerDied","Data":"4ce5eb7249d07f9d5a8bae5c243853926fec914f70cc866310d8cb349ee22ed1"}
Oct 30 22:52:14.576842 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.576808    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"]
Oct 30 22:52:14.580923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.580904    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.637288 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.637259    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"]
Oct 30 22:52:14.745884 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.745848    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.746043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.745894    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.746043 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.745923    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwc9\" (UniqueName: \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.847404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.847325    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.847404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.847367    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.847404 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.847383    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwc9\" (UniqueName: \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.848127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.848105    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.849850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.849827    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:14.903863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:14.903836    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwc9\" (UniqueName: \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:15.195495 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.195463    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:15.387263 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.387238    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:15.454084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.454027    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\" (UniqueName: \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\") pod \"fd5798df-3433-4aa9-8801-58f0d34e340f\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") "
Oct 30 22:52:15.454220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.454122    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\") pod \"fd5798df-3433-4aa9-8801-58f0d34e340f\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") "
Oct 30 22:52:15.454220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.454141    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\") pod \"fd5798df-3433-4aa9-8801-58f0d34e340f\" (UID: \"fd5798df-3433-4aa9-8801-58f0d34e340f\") "
Oct 30 22:52:15.454537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.454515    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca" pod "fd5798df-3433-4aa9-8801-58f0d34e340f" (UID: "fd5798df-3433-4aa9-8801-58f0d34e340f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:15.455999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.455976    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx" pod "fd5798df-3433-4aa9-8801-58f0d34e340f" (UID: "fd5798df-3433-4aa9-8801-58f0d34e340f"). InnerVolumeSpecName "kube-api-access-bv8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:15.456082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.456058    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca" pod "fd5798df-3433-4aa9-8801-58f0d34e340f" (UID: "fd5798df-3433-4aa9-8801-58f0d34e340f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:15.555974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.555941    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:15.555974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.555971    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5798df-3433-4aa9-8801-58f0d34e340f-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:15.555974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.555981    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8gx\" (UniqueName: \"kubernetes.io/projected/fd5798df-3433-4aa9-8801-58f0d34e340f-kube-api-access-bv8gx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:15.571737 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:52:15.571700    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1835cd9b_58fd_4fb0_b471_bc5d10d22a2e.slice/crio-aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887 WatchSource:0}: Error finding container aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887: Status 404 returned error can't find the container with id aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887
Oct 30 22:52:15.575338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:15.575311    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"]
Oct 30 22:52:16.252869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.252831    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv" event={"ID":"fd5798df-3433-4aa9-8801-58f0d34e340f","Type":"ContainerDied","Data":"03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5"}
Oct 30 22:52:16.252869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.252865    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"
Oct 30 22:52:16.253407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.252890    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b0d3c7c103fcb1e63fdf0793981268d697576ca3577c57c0d17611f31413d5"
Oct 30 22:52:16.254833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.254802    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq" event={"ID":"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e","Type":"ContainerStarted","Data":"ee4e4d6e50add782c9aedf6ef36f36e93dfeb324d17a6cefe3cdf744cde22e5d"}
Oct 30 22:52:16.254977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.254855    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq" event={"ID":"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e","Type":"ContainerStarted","Data":"aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887"}
Oct 30 22:52:16.310452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.310427    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:16.368113 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.368060    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq" podStartSLOduration=2.3680463339999998 podStartE2EDuration="2.368046334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:16.366269699 +0000 UTC m=+4189.455007501" watchObservedRunningTime="2026-10-30 22:52:16.368046334 +0000 UTC m=+4189.456784186"
Oct 30 22:52:16.495280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:16.495238    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:17.262232 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:17.262198    4410 generic.go:356] "Generic (PLEG): container finished" podID="1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" containerID="ee4e4d6e50add782c9aedf6ef36f36e93dfeb324d17a6cefe3cdf744cde22e5d" exitCode=0
Oct 30 22:52:17.262641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:17.262296    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq" event={"ID":"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e","Type":"ContainerDied","Data":"ee4e4d6e50add782c9aedf6ef36f36e93dfeb324d17a6cefe3cdf744cde22e5d"}
Oct 30 22:52:17.669526 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:17.669492    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:18.393357 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.393332    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:18.486417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.486384    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\") pod \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") "
Oct 30 22:52:18.486621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.486509    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\" (UniqueName: \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\") pod \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") "
Oct 30 22:52:18.486621 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.486549    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\") pod \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\" (UID: \"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e\") "
Oct 30 22:52:18.486915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.486892    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca" pod "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" (UID: "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:18.488475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.488455    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca" pod "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" (UID: "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:18.488542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.488519    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9" pod "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" (UID: "1835cd9b-58fd-4fb0-b471-bc5d10d22a2e"). InnerVolumeSpecName "kube-api-access-7rwc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:18.588265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.588186    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwc9\" (UniqueName: \"kubernetes.io/projected/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-kube-api-access-7rwc9\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:18.588265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.588216    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:18.588265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.588227    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:18.896465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:18.896434    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.276267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:19.276172    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq" event={"ID":"1835cd9b-58fd-4fb0-b471-bc5d10d22a2e","Type":"ContainerDied","Data":"aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887"}
Oct 30 22:52:19.276267 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:19.276258    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec6f656dbc4d844654180c0854f258c5537f8fcd4e5f048fa0bfd8c344c0887"
Oct 30 22:52:19.276457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:19.276186    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"
Oct 30 22:52:19.463050 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:19.463005    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:20.521221 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:52:20.521186    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:52:20.667013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:20.666960    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:21.000007 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:20.999973    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:21.562347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:21.562315    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:23.739860 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:23.739808    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:52:24.153869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:24.153834    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:24.378820 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:24.378782    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.119516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.119480    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"]
Oct 30 22:52:26.119931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.119917    4410 cpu_manager.go:414] "containerMap: removing container" podUID="1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" containerName="revoke-keys"
Oct 30 22:52:26.119980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.119932    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" containerName="revoke-keys"
Oct 30 22:52:26.119980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.119967    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fd5798df-3433-4aa9-8801-58f0d34e340f" containerName="revoke-keys"
Oct 30 22:52:26.119980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.119972    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5798df-3433-4aa9-8801-58f0d34e340f" containerName="revoke-keys"
Oct 30 22:52:26.120086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.120029    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5798df-3433-4aa9-8801-58f0d34e340f" containerName="revoke-keys"
Oct 30 22:52:26.120086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.120041    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" containerName="revoke-keys"
Oct 30 22:52:26.123374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.123358    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.162402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.162364    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.162543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.162407    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9fx\" (UniqueName: \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.162543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.162479    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.176384 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.176362    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:26.225879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.225843    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:26.225879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.225860    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:52:26.225879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.225873    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:26.228373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.228347    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"]
Oct 30 22:52:26.230467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.230448    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"]
Oct 30 22:52:26.234162 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.234147    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.263351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263323    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.263467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263393    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cq9\" (UniqueName: \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.263514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263462    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.263514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263492    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9fx\" (UniqueName: \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.263601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263519    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.263601 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.263550    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.264107 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.264087    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.265923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.265901    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.292833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.292804    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"]
Oct 30 22:52:26.321488 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.321463    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9fx\" (UniqueName: \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\") pod \"maas-api-revoke-sub-7bbbbeae-tgbvj\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.364599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.364566    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.364768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.364611    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cq9\" (UniqueName: \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.364768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.364651    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.365355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.365328    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.366899 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.366870    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.374595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.374530    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.428233 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.428205    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cq9\" (UniqueName: \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\") pod \"maas-api-revoke-sub-4d4eaf46-s9lhk\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.436548 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.436528    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:26.548200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.548163    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:26.638271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.638124    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.806418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.806387    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.814544 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.814522    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"]
Oct 30 22:52:26.817234 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:52:26.817208    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b124ed_1e81_4a99_ad73_105af27af79d.slice/crio-d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39 WatchSource:0}: Error finding container d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39: Status 404 returned error can't find the container with id d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39
Oct 30 22:52:26.934615 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:26.934587    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"]
Oct 30 22:52:26.936341 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:52:26.936311    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0fc698_3975_4f41_9394_18d65c920574.slice/crio-e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71 WatchSource:0}: Error finding container e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71: Status 404 returned error can't find the container with id e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71
Oct 30 22:52:27.361838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:27.337940    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk" event={"ID":"6d0fc698-3975-4f41-9394-18d65c920574","Type":"ContainerStarted","Data":"e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71"}
Oct 30 22:52:27.361838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:27.339971    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj" event={"ID":"f2b124ed-1e81-4a99-ad73-105af27af79d","Type":"ContainerStarted","Data":"d1d101b3667bb816f1e896e8581846c203dc19b3f30ae9bb78b7aa558bdc1eaf"}
Oct 30 22:52:27.361838 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:27.340043    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj" event={"ID":"f2b124ed-1e81-4a99-ad73-105af27af79d","Type":"ContainerStarted","Data":"d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39"}
Oct 30 22:52:27.456139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:27.456087    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj" podStartSLOduration=1.456072614 podStartE2EDuration="1.456072614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:27.45430615 +0000 UTC m=+4200.543043954" watchObservedRunningTime="2026-10-30 22:52:27.456072614 +0000 UTC m=+4200.544810417"
Oct 30 22:52:28.347302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.347267    4410 generic.go:356] "Generic (PLEG): container finished" podID="6d0fc698-3975-4f41-9394-18d65c920574" containerID="e9bf8272c3033e52d2844e45a55982ebc1f4a97e87264b730759fda153b83a77" exitCode=0
Oct 30 22:52:28.347489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.347355    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk" event={"ID":"6d0fc698-3975-4f41-9394-18d65c920574","Type":"ContainerDied","Data":"e9bf8272c3033e52d2844e45a55982ebc1f4a97e87264b730759fda153b83a77"}
Oct 30 22:52:28.348879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.348854    4410 generic.go:356] "Generic (PLEG): container finished" podID="f2b124ed-1e81-4a99-ad73-105af27af79d" containerID="d1d101b3667bb816f1e896e8581846c203dc19b3f30ae9bb78b7aa558bdc1eaf" exitCode=0
Oct 30 22:52:28.349011 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.348933    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj" event={"ID":"f2b124ed-1e81-4a99-ad73-105af27af79d","Type":"ContainerDied","Data":"d1d101b3667bb816f1e896e8581846c203dc19b3f30ae9bb78b7aa558bdc1eaf"}
Oct 30 22:52:28.379599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.379568    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.741067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:28.741026    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing" containerID="cri-o://d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7" gracePeriod=25
Oct 30 22:52:29.518323 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.518293    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:29.521293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.521274    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:29.610181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610128    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\") pod \"f2b124ed-1e81-4a99-ad73-105af27af79d\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") "
Oct 30 22:52:29.610181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610168    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\") pod \"6d0fc698-3975-4f41-9394-18d65c920574\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") "
Oct 30 22:52:29.610365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610195    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\" (UniqueName: \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\") pod \"6d0fc698-3975-4f41-9394-18d65c920574\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") "
Oct 30 22:52:29.610365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610313    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\" (UniqueName: \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\") pod \"f2b124ed-1e81-4a99-ad73-105af27af79d\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") "
Oct 30 22:52:29.610365 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610347    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\") pod \"6d0fc698-3975-4f41-9394-18d65c920574\" (UID: \"6d0fc698-3975-4f41-9394-18d65c920574\") "
Oct 30 22:52:29.610553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610385    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\") pod \"f2b124ed-1e81-4a99-ad73-105af27af79d\" (UID: \"f2b124ed-1e81-4a99-ad73-105af27af79d\") "
Oct 30 22:52:29.610553 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610540    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca" pod "f2b124ed-1e81-4a99-ad73-105af27af79d" (UID: "f2b124ed-1e81-4a99-ad73-105af27af79d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:29.610662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610540    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca" pod "6d0fc698-3975-4f41-9394-18d65c920574" (UID: "6d0fc698-3975-4f41-9394-18d65c920574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:29.610762 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610745    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.610852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.610770    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.612420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.612393    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca" pod "6d0fc698-3975-4f41-9394-18d65c920574" (UID: "6d0fc698-3975-4f41-9394-18d65c920574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:29.612420 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.612402    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9" pod "6d0fc698-3975-4f41-9394-18d65c920574" (UID: "6d0fc698-3975-4f41-9394-18d65c920574"). InnerVolumeSpecName "kube-api-access-k8cq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:29.612551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.612484    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca" pod "f2b124ed-1e81-4a99-ad73-105af27af79d" (UID: "f2b124ed-1e81-4a99-ad73-105af27af79d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:29.612815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.612767    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx" pod "f2b124ed-1e81-4a99-ad73-105af27af79d" (UID: "f2b124ed-1e81-4a99-ad73-105af27af79d"). InnerVolumeSpecName "kube-api-access-xs9fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:29.712132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.712101    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9fx\" (UniqueName: \"kubernetes.io/projected/f2b124ed-1e81-4a99-ad73-105af27af79d-kube-api-access-xs9fx\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.712132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.712129    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0fc698-3975-4f41-9394-18d65c920574-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.712132 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.712140    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b124ed-1e81-4a99-ad73-105af27af79d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.712361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:29.712148    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8cq9\" (UniqueName: \"kubernetes.io/projected/6d0fc698-3975-4f41-9394-18d65c920574-kube-api-access-k8cq9\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:30.361563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.361531    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"
Oct 30 22:52:30.361743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.361529    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk" event={"ID":"6d0fc698-3975-4f41-9394-18d65c920574","Type":"ContainerDied","Data":"e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71"}
Oct 30 22:52:30.361743 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.361649    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6e2c1d717c445f8f9b7e9350aaf0939878d358008215e29bc16bd1508549b71"
Oct 30 22:52:30.362958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.362939    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"
Oct 30 22:52:30.363078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.362987    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj" event={"ID":"f2b124ed-1e81-4a99-ad73-105af27af79d","Type":"ContainerDied","Data":"d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39"}
Oct 30 22:52:30.363078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.363029    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d93fbe29dcba440089d90bab2e39bae18d02196529a63b4a74eede0cffe1ce39"
Oct 30 22:52:30.763652 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:30.763620    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:31.016094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:31.016006    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:31.260383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:31.260350    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"]
Oct 30 22:52:31.309515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:31.309445    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b1eb3f-z9xzq"]
Oct 30 22:52:31.467555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:31.467515    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:31.675573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:31.675538    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:33.018430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:33.018400    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1835cd9b-58fd-4fb0-b471-bc5d10d22a2e" path="/var/lib/kubelet/pods/1835cd9b-58fd-4fb0-b471-bc5d10d22a2e/volumes"
Oct 30 22:52:33.705523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:33.705487    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.612375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:37.612341    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.786547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:37.786513    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:38.021745 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.021702    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:38.187418 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.187374    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:38.445039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.445009    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:38.988558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.988522    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"]
Oct 30 22:52:38.989014 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.988998    4410 cpu_manager.go:414] "containerMap: removing container" podUID="6d0fc698-3975-4f41-9394-18d65c920574" containerName="revoke-keys"
Oct 30 22:52:38.989102 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.989014    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0fc698-3975-4f41-9394-18d65c920574" containerName="revoke-keys"
Oct 30 22:52:38.989138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.989129    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f2b124ed-1e81-4a99-ad73-105af27af79d" containerName="revoke-keys"
Oct 30 22:52:38.989176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.989136    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b124ed-1e81-4a99-ad73-105af27af79d" containerName="revoke-keys"
Oct 30 22:52:38.989226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.989209    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0fc698-3975-4f41-9394-18d65c920574" containerName="revoke-keys"
Oct 30 22:52:38.989226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.989224    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b124ed-1e81-4a99-ad73-105af27af79d" containerName="revoke-keys"
Oct 30 22:52:38.992654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:38.992625    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.090940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.090907    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:39.090940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.090909    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:39.091170 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.091045    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:39.095165 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.095080    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"]
Oct 30 22:52:39.096348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.096320    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:52:39.106787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.106742    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.107002 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.106971    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8lq\" (UniqueName: \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.107118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.107095    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.208564 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.208538    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.208728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.208574    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8lq\" (UniqueName: \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.208728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.208629    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.209227 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.209208    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.210866 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.210846    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.264628 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.264573    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8lq\" (UniqueName: \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.309442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.309417    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:39.678660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.678637    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"]
Oct 30 22:52:39.680861 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:52:39.680833    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835dde92_eb75_4747_8618_a50a6058406c.slice/crio-266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a WatchSource:0}: Error finding container 266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a: Status 404 returned error can't find the container with id 266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a
Oct 30 22:52:39.776084 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:39.776055    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.095590 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.095556    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.252979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.252897    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.306599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.306571    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"]
Oct 30 22:52:40.308667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.308643    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1c3eef-62rvb"]
Oct 30 22:52:40.419075 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.419042    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g" event={"ID":"835dde92-eb75-4747-8618-a50a6058406c","Type":"ContainerStarted","Data":"57f864d2deafcaa1d58c166ac9b10895b55de228e6ae2b599af9332e88136c97"}
Oct 30 22:52:40.419223 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.419091    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g" event={"ID":"835dde92-eb75-4747-8618-a50a6058406c","Type":"ContainerStarted","Data":"266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a"}
Oct 30 22:52:40.536537 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.536440    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g" podStartSLOduration=2.5364257759999997 podStartE2EDuration="2.536425776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:40.53337957 +0000 UTC m=+4213.622117372" watchObservedRunningTime="2026-10-30 22:52:40.536425776 +0000 UTC m=+4213.625163578"
Oct 30 22:52:40.749709 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:40.749664    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:41.018134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:41.018099    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e3e437-ecf8-4eca-a25b-ea402ca78749" path="/var/lib/kubelet/pods/d7e3e437-ecf8-4eca-a25b-ea402ca78749/volumes"
Oct 30 22:52:41.433964 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:41.433930    4410 generic.go:356] "Generic (PLEG): container finished" podID="835dde92-eb75-4747-8618-a50a6058406c" containerID="57f864d2deafcaa1d58c166ac9b10895b55de228e6ae2b599af9332e88136c97" exitCode=0
Oct 30 22:52:41.434359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:41.434012    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g" event={"ID":"835dde92-eb75-4747-8618-a50a6058406c","Type":"ContainerDied","Data":"57f864d2deafcaa1d58c166ac9b10895b55de228e6ae2b599af9332e88136c97"}
Oct 30 22:52:41.468220 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:41.468190    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:42.577294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.577268    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:42.744362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.744276    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\") pod \"835dde92-eb75-4747-8618-a50a6058406c\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") "
Oct 30 22:52:42.744362 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.744329    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\" (UniqueName: \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\") pod \"835dde92-eb75-4747-8618-a50a6058406c\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") "
Oct 30 22:52:42.744571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.744465    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\") pod \"835dde92-eb75-4747-8618-a50a6058406c\" (UID: \"835dde92-eb75-4747-8618-a50a6058406c\") "
Oct 30 22:52:42.744716 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.744690    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca" pod "835dde92-eb75-4747-8618-a50a6058406c" (UID: "835dde92-eb75-4747-8618-a50a6058406c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:42.744849 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.744831    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835dde92-eb75-4747-8618-a50a6058406c-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:42.746466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.746433    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca" pod "835dde92-eb75-4747-8618-a50a6058406c" (UID: "835dde92-eb75-4747-8618-a50a6058406c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:42.746466 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.746455    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq" pod "835dde92-eb75-4747-8618-a50a6058406c" (UID: "835dde92-eb75-4747-8618-a50a6058406c"). InnerVolumeSpecName "kube-api-access-7r8lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:42.846256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.846222    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8lq\" (UniqueName: \"kubernetes.io/projected/835dde92-eb75-4747-8618-a50a6058406c-kube-api-access-7r8lq\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:42.846256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:42.846251    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835dde92-eb75-4747-8618-a50a6058406c-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:43.231127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.231095    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.429393 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.429361    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 22:52:43.448656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.448619    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g" event={"ID":"835dde92-eb75-4747-8618-a50a6058406c","Type":"ContainerDied","Data":"266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a"}
Oct 30 22:52:43.448836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.448676    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="266485a6c805ab7986f77255aab240a8caec87d87070dbeb09b4c10e0162980a"
Oct 30 22:52:43.448836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.448640    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"
Oct 30 22:52:43.486133 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.486096    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.662297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:43.662264    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.438627 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:45.438591    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:46.460683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:46.460651    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:46.628428 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:46.628397    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:47.029802 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:47.029741    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:47.294529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:47.294426    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:47.369387 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:47.369348    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:52:47.369610 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:47.369584    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" containerID="cri-o://700b66ff98b3f054322bc1289632f5f9d30d08507c19927806ec27e03885243d" gracePeriod=30
Oct 30 22:52:47.483242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:47.483208    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:48.996514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:48.996490    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:52:49.005004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.004949    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\") pod \"2eba5e00-f84a-4644-ab13-601b2c57fffd\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") "
Oct 30 22:52:49.005137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.005079    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\" (UniqueName: \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\") pod \"2eba5e00-f84a-4644-ab13-601b2c57fffd\" (UID: \"2eba5e00-f84a-4644-ab13-601b2c57fffd\") "
Oct 30 22:52:49.005330 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.005308    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume" pod "2eba5e00-f84a-4644-ab13-601b2c57fffd" (UID: "2eba5e00-f84a-4644-ab13-601b2c57fffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:49.007078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.007059    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb" pod "2eba5e00-f84a-4644-ab13-601b2c57fffd" (UID: "2eba5e00-f84a-4644-ab13-601b2c57fffd"). InnerVolumeSpecName "kube-api-access-tthjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:49.106506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.106476    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tthjb\" (UniqueName: \"kubernetes.io/projected/2eba5e00-f84a-4644-ab13-601b2c57fffd-kube-api-access-tthjb\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.106506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.106502    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eba5e00-f84a-4644-ab13-601b2c57fffd-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.482965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.482925    4410 generic.go:356] "Generic (PLEG): container finished" podID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerID="d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7" exitCode=0
Oct 30 22:52:49.483138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.482972    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"
Oct 30 22:52:49.483138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.483012    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" event={"ID":"2eba5e00-f84a-4644-ab13-601b2c57fffd","Type":"ContainerDied","Data":"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"}
Oct 30 22:52:49.483138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.483068    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm" event={"ID":"2eba5e00-f84a-4644-ab13-601b2c57fffd","Type":"ContainerDied","Data":"5e12f17bc675287e42d19e0a61e1a0f6caada51fbdace0f2fdb1d84421aed7b9"}
Oct 30 22:52:49.483138 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.483097    4410 scope.go:122] "RemoveContainer" containerID="d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"
Oct 30 22:52:49.492687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.492668    4410 scope.go:122] "RemoveContainer" containerID="d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"
Oct 30 22:52:49.492965 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:52:49.492946    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7\": container with ID starting with d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7 not found: ID does not exist" containerID="d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"
Oct 30 22:52:49.493049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.492978    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7"} err="failed to get container status \"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7\": rpc error: code = NotFound desc = could not find container \"d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7\": container with ID starting with d666d0ffce9940fc5a802c71566c7dc5931d9616712209e1b58b77c90f1e36c7 not found: ID does not exist"
Oct 30 22:52:49.645078 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.645049    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:52:49.650515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:49.650493    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c788d-d46484bf9-wp8nm"]
Oct 30 22:52:50.125826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:50.125752    4410 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:52:50.126248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:50.125856    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:52:50.377630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:50.377542    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.525155 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:52:50.525079    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:52:50.527076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:50.527049    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.796423 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:50.796388    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:51.018595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:51.018561    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" path="/var/lib/kubelet/pods/2eba5e00-f84a-4644-ab13-601b2c57fffd/volumes"
Oct 30 22:52:52.503728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.503699    4410 generic.go:356] "Generic (PLEG): container finished" podID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerID="700b66ff98b3f054322bc1289632f5f9d30d08507c19927806ec27e03885243d" exitCode=0
Oct 30 22:52:52.504143 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.503795    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" event={"ID":"c6a4c55f-2574-49aa-a3c5-17d775c86e8c","Type":"ContainerDied","Data":"700b66ff98b3f054322bc1289632f5f9d30d08507c19927806ec27e03885243d"}
Oct 30 22:52:52.616065 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.616037    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:52:52.640306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640284    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640328    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640370    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640407 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640390    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640421    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640455    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640498    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:52.640560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640546    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640577    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640617    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:52.640799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640626    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\") pod \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\" (UID: \"c6a4c55f-2574-49aa-a3c5-17d775c86e8c\") "
Oct 30 22:52:52.640799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640728    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:52.641036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.640881    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:52.641090 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641039    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:52.641157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641136    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.641200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641163    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.641200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641178    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.641200 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641192    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.641596 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.641205    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.642686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.642659    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:52:52.642686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.642674    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "kube-api-access-xhphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:52.643077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.643049    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:52.643077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.643066    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token" pod "c6a4c55f-2574-49aa-a3c5-17d775c86e8c" (UID: "c6a4c55f-2574-49aa-a3c5-17d775c86e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:52.741670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.741643    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.741834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.741686    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhphj\" (UniqueName: \"kubernetes.io/projected/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-kube-api-access-xhphj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.741834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.741697    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.741834 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:52.741706    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a4c55f-2574-49aa-a3c5-17d775c86e8c-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:53.511131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:53.511089    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh" event={"ID":"c6a4c55f-2574-49aa-a3c5-17d775c86e8c","Type":"ContainerDied","Data":"033fc558c5a915d82c957fd5325b39195a82f8ae38d0f215cd2f34dd658bbff6"}
Oct 30 22:52:53.511131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:53.511119    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"
Oct 30 22:52:53.511589 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:53.511151    4410 scope.go:122] "RemoveContainer" containerID="700b66ff98b3f054322bc1289632f5f9d30d08507c19927806ec27e03885243d"
Oct 30 22:52:53.723594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:53.723563    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:52:53.766636 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:53.766572    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-whbjh"]
Oct 30 22:52:55.018147 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:55.018112    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" path="/var/lib/kubelet/pods/c6a4c55f-2574-49aa-a3c5-17d775c86e8c/volumes"
Oct 30 22:52:55.301998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:55.301919    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:55.464943 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:55.464911    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:55.636470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:55.636380    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:58.449940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.449891    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" containerID="cri-o://862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5" gracePeriod=45
Oct 30 22:52:58.713020 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.712961    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:52:58.798826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.798794    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.798974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.798857    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.798974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.798883    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.798974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.798917    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\" (UniqueName: \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.799161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.798991    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.799161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799019    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.799161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799088    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\" (UniqueName: \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\") pod \"cac9b08b-7aa9-401c-87e4-8be2eef06427\" (UID: \"cac9b08b-7aa9-401c-87e4-8be2eef06427\") "
Oct 30 22:52:58.799161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799151    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.799354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799280    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.799415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799397    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.799452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799420    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.799452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.799447    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.801096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.801073    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.801183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.801169    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "kube-api-access-t6ms6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:58.801221 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.801178    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs" pod "cac9b08b-7aa9-401c-87e4-8be2eef06427" (UID: "cac9b08b-7aa9-401c-87e4-8be2eef06427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:58.900695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.900655    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac9b08b-7aa9-401c-87e4-8be2eef06427-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.900695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.900696    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ms6\" (UniqueName: \"kubernetes.io/projected/cac9b08b-7aa9-401c-87e4-8be2eef06427-kube-api-access-t6ms6\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.900981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:58.900712    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac9b08b-7aa9-401c-87e4-8be2eef06427-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:59.546904 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.546870    4410 generic.go:356] "Generic (PLEG): container finished" podID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerID="862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5" exitCode=0
Oct 30 22:52:59.547449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.546927    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"
Oct 30 22:52:59.547449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.546964    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" event={"ID":"cac9b08b-7aa9-401c-87e4-8be2eef06427","Type":"ContainerDied","Data":"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"}
Oct 30 22:52:59.547449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.547011    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" event={"ID":"cac9b08b-7aa9-401c-87e4-8be2eef06427","Type":"ContainerDied","Data":"ca10961dd95c87e63ba94bd3a0b04d027f3a83086797bfad1652f77ae7d31401"}
Oct 30 22:52:59.547449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.547032    4410 scope.go:122] "RemoveContainer" containerID="862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"
Oct 30 22:52:59.556659 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.556644    4410 scope.go:122] "RemoveContainer" containerID="862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"
Oct 30 22:52:59.556939 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:52:59.556922    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5\": container with ID starting with 862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5 not found: ID does not exist" containerID="862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"
Oct 30 22:52:59.556997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.556949    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5"} err="failed to get container status \"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5\": rpc error: code = NotFound desc = could not find container \"862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5\": container with ID starting with 862322c3b949773c023c99fdb734310c18099dce7ee6f45bbaf5cf712472b9c5 not found: ID does not exist"
Oct 30 22:52:59.615508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.615480    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:52:59.709236 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.709203    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 22:52:59.712551 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:52:59.712526    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/simulator-w0-b8717e-kserve-7c4bfff749-rzdxd"]
Oct 30 22:53:00.441086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:00.441057    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.018446 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:01.018412    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" path="/var/lib/kubelet/pods/cac9b08b-7aa9-401c-87e4-8be2eef06427/volumes"
Oct 30 22:53:04.659878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:04.659836    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:05.810470 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:05.810434    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:06.176833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:06.176765    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:06.455609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:06.455530    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:06.656665 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:06.656626    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"]
Oct 30 22:53:06.704361 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:06.704331    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c788d-2dhdv"]
Oct 30 22:53:07.018869 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.018835    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5798df-3433-4aa9-8801-58f0d34e340f" path="/var/lib/kubelet/pods/fd5798df-3433-4aa9-8801-58f0d34e340f/volumes"
Oct 30 22:53:07.079174 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.079144    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.413878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.413844    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.702569 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.702487    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.867289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.867253    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.948828 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.948800    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"]
Oct 30 22:53:07.998260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:07.998194    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d2a3e0-m2x45"]
Oct 30 22:53:08.388451 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:08.388408    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:09.019276 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:09.019245    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bde9c9-8029-4f63-8ff5-e9c0b4b551c5" path="/var/lib/kubelet/pods/54bde9c9-8029-4f63-8ff5-e9c0b4b551c5/volumes"
Oct 30 22:53:12.337514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337475    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"]
Oct 30 22:53:12.337928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337897    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy"
Oct 30 22:53:12.337928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337913    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337929    4410 cpu_manager.go:414] "containerMap: removing container" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337937    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337948    4410 cpu_manager.go:414] "containerMap: removing container" podUID="835dde92-eb75-4747-8618-a50a6058406c" containerName="revoke-keys"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337953    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835dde92-eb75-4747-8618-a50a6058406c" containerName="revoke-keys"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337961    4410 cpu_manager.go:414] "containerMap: removing container" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main"
Oct 30 22:53:12.338017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.337965    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main"
Oct 30 22:53:12.338202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.338039    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eba5e00-f84a-4644-ab13-601b2c57fffd" containerName="payload-processing"
Oct 30 22:53:12.338202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.338049    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="835dde92-eb75-4747-8618-a50a6058406c" containerName="revoke-keys"
Oct 30 22:53:12.338202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.338058    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac9b08b-7aa9-401c-87e4-8be2eef06427" containerName="main"
Oct 30 22:53:12.338202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.338075    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a4c55f-2574-49aa-a3c5-17d775c86e8c" containerName="istio-proxy"
Oct 30 22:53:12.341306 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.341289    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.393081 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.393056    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"]
Oct 30 22:53:12.393533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.393516    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:12.396609 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.396591    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.428502 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.428476    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.428622 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.428507    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpwm\" (UniqueName: \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.428688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.428639    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.440178 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.440155    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:12.440312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.440194    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:12.440312 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.440245    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:53:12.442577 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.442528    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"]
Oct 30 22:53:12.455261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.455232    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"]
Oct 30 22:53:12.530260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530235    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.530399 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530283    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.530447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530425    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chsrm\" (UniqueName: \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.530499 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530486    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.530555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530544    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.530631 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530615    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpwm\" (UniqueName: \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.530920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.530902    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.532823 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.532804    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.592412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.592346    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpwm\" (UniqueName: \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\") pod \"maas-api-revoke-sub-069c851f-pwkdr\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.632166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.632140    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.632287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.632204    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chsrm\" (UniqueName: \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.632287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.632236    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.632941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.632919    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.634271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.634254    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.654062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.654040    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:12.693645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.693616    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chsrm\" (UniqueName: \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\") pod \"maas-api-revoke-sub-99492467-ffstm\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:12.708131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:12.707986    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:13.025997 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.025972    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"]
Oct 30 22:53:13.027900 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:53:13.027868    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c91f42_2504_4749_8cef_8b5e161a9580.slice/crio-19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad WatchSource:0}: Error finding container 19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad: Status 404 returned error can't find the container with id 19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad
Oct 30 22:53:13.084975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.084948    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"]
Oct 30 22:53:13.087286 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:53:13.087262    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1cc120_6383_493a_adc0_16ae80ed2215.slice/crio-2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7 WatchSource:0}: Error finding container 2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7: Status 404 returned error can't find the container with id 2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7
Oct 30 22:53:13.298135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.298080    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:13.581511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.581437    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:13.640808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.630267    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm" event={"ID":"fc1cc120-6383-493a-adc0-16ae80ed2215","Type":"ContainerStarted","Data":"3bd7983757be351a59df37b63fab1ecc1976276aeaf23d6095be71a402c15f4a"}
Oct 30 22:53:13.640808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.630335    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm" event={"ID":"fc1cc120-6383-493a-adc0-16ae80ed2215","Type":"ContainerStarted","Data":"2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7"}
Oct 30 22:53:13.640808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.632262    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr" event={"ID":"00c91f42-2504-4749-8cef-8b5e161a9580","Type":"ContainerStarted","Data":"431d94baa61f8ede05e87df22cb9ecbc8fe47ab35fd0b438997e84780fc2968f"}
Oct 30 22:53:13.640808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.632305    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr" event={"ID":"00c91f42-2504-4749-8cef-8b5e161a9580","Type":"ContainerStarted","Data":"19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad"}
Oct 30 22:53:13.745119 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.745064    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm" podStartSLOduration=1.7450521430000001 podStartE2EDuration="1.745052143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:13.743017815 +0000 UTC m=+4246.831755628" watchObservedRunningTime="2026-10-30 22:53:13.745052143 +0000 UTC m=+4246.833789945"
Oct 30 22:53:13.769477 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.769446    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:13.857593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:13.857507    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr" podStartSLOduration=1.8574916830000001 podStartE2EDuration="1.857491683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:13.855951255 +0000 UTC m=+4246.944689072" watchObservedRunningTime="2026-10-30 22:53:13.857491683 +0000 UTC m=+4246.946229486"
Oct 30 22:53:14.639096 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:14.639062    4410 generic.go:356] "Generic (PLEG): container finished" podID="fc1cc120-6383-493a-adc0-16ae80ed2215" containerID="3bd7983757be351a59df37b63fab1ecc1976276aeaf23d6095be71a402c15f4a" exitCode=0
Oct 30 22:53:14.639482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:14.639147    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm" event={"ID":"fc1cc120-6383-493a-adc0-16ae80ed2215","Type":"ContainerDied","Data":"3bd7983757be351a59df37b63fab1ecc1976276aeaf23d6095be71a402c15f4a"}
Oct 30 22:53:14.640662 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:14.640638    4410 generic.go:356] "Generic (PLEG): container finished" podID="00c91f42-2504-4749-8cef-8b5e161a9580" containerID="431d94baa61f8ede05e87df22cb9ecbc8fe47ab35fd0b438997e84780fc2968f" exitCode=0
Oct 30 22:53:14.640811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:14.640669    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr" event={"ID":"00c91f42-2504-4749-8cef-8b5e161a9580","Type":"ContainerDied","Data":"431d94baa61f8ede05e87df22cb9ecbc8fe47ab35fd0b438997e84780fc2968f"}
Oct 30 22:53:15.162541 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.162510    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:15.316672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.316637    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:15.465197 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.465112    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:53:15.465356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.465309    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy" containerID="cri-o://819e48a54ba46336767962644b5937f7f0a6e5ad28e5c24751f44a62db428327" gracePeriod=30
Oct 30 22:53:15.491116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.491081    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:15.786938 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.786911    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:15.815329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.815305    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:15.864977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.864944    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\") pod \"fc1cc120-6383-493a-adc0-16ae80ed2215\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") "
Oct 30 22:53:15.865148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.864999    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\") pod \"fc1cc120-6383-493a-adc0-16ae80ed2215\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") "
Oct 30 22:53:15.865148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.865135    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\" (UniqueName: \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\") pod \"fc1cc120-6383-493a-adc0-16ae80ed2215\" (UID: \"fc1cc120-6383-493a-adc0-16ae80ed2215\") "
Oct 30 22:53:15.865315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.865293    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca" pod "fc1cc120-6383-493a-adc0-16ae80ed2215" (UID: "fc1cc120-6383-493a-adc0-16ae80ed2215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:15.865429 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.865412    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:15.867140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.867113    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca" pod "fc1cc120-6383-493a-adc0-16ae80ed2215" (UID: "fc1cc120-6383-493a-adc0-16ae80ed2215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:15.867140 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.867124    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm" pod "fc1cc120-6383-493a-adc0-16ae80ed2215" (UID: "fc1cc120-6383-493a-adc0-16ae80ed2215"). InnerVolumeSpecName "kube-api-access-chsrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:15.966139 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966114    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\") pod \"00c91f42-2504-4749-8cef-8b5e161a9580\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") "
Oct 30 22:53:15.966302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966223    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\") pod \"00c91f42-2504-4749-8cef-8b5e161a9580\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") "
Oct 30 22:53:15.966302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966299    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\" (UniqueName: \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\") pod \"00c91f42-2504-4749-8cef-8b5e161a9580\" (UID: \"00c91f42-2504-4749-8cef-8b5e161a9580\") "
Oct 30 22:53:15.966560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966533    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca" pod "00c91f42-2504-4749-8cef-8b5e161a9580" (UID: "00c91f42-2504-4749-8cef-8b5e161a9580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:15.966560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966544    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chsrm\" (UniqueName: \"kubernetes.io/projected/fc1cc120-6383-493a-adc0-16ae80ed2215-kube-api-access-chsrm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:15.966684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.966577    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1cc120-6383-493a-adc0-16ae80ed2215-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:15.968310 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.968288    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca" pod "00c91f42-2504-4749-8cef-8b5e161a9580" (UID: "00c91f42-2504-4749-8cef-8b5e161a9580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:15.968414 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:15.968364    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm" pod "00c91f42-2504-4749-8cef-8b5e161a9580" (UID: "00c91f42-2504-4749-8cef-8b5e161a9580"). InnerVolumeSpecName "kube-api-access-vkpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:16.067185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.067114    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpwm\" (UniqueName: \"kubernetes.io/projected/00c91f42-2504-4749-8cef-8b5e161a9580-kube-api-access-vkpwm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.067185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.067139    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.067185 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.067150    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c91f42-2504-4749-8cef-8b5e161a9580-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.653681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.653649    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"
Oct 30 22:53:16.653681 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.653674    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm" event={"ID":"fc1cc120-6383-493a-adc0-16ae80ed2215","Type":"ContainerDied","Data":"2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7"}
Oct 30 22:53:16.653937 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.653727    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eebe62558ec4f786fe0de600f7199b3d7f909e5e7c91239e421116f6fabe3f7"
Oct 30 22:53:16.655067 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.655050    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"
Oct 30 22:53:16.655176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.655092    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr" event={"ID":"00c91f42-2504-4749-8cef-8b5e161a9580","Type":"ContainerDied","Data":"19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad"}
Oct 30 22:53:16.655176 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:16.655142    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19420461a2c66df9a6a4c7d199fa6647207a5c8c5d52934b0f3ffa4d5c16d9ad"
Oct 30 22:53:18.709467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:18.709435    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:18.880025 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:18.879995    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:19.706562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:19.706523    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:19.880854 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:19.880823    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:20.521460 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:53:20.521427    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:53:20.687489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.687216    4410 generic.go:356] "Generic (PLEG): container finished" podID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerID="819e48a54ba46336767962644b5937f7f0a6e5ad28e5c24751f44a62db428327" exitCode=0
Oct 30 22:53:20.687489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.687422    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" event={"ID":"e703d5a0-297b-4e94-8686-4a490c09cd22","Type":"ContainerDied","Data":"819e48a54ba46336767962644b5937f7f0a6e5ad28e5c24751f44a62db428327"}
Oct 30 22:53:20.728638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.728615    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:53:20.813302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813228    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813267    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813291    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813333    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813371    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813422    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813456    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813485    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.813595 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813548    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\") pod \"e703d5a0-297b-4e94-8686-4a490c09cd22\" (UID: \"e703d5a0-297b-4e94-8686-4a490c09cd22\") "
Oct 30 22:53:20.814017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813693    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:20.814017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.813713    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:20.814017 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.814002    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.814188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.814026    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.814188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.814103    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:20.814188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.814133    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:20.814188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.814169    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:20.815606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.815579    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:20.815606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.815597    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:53:20.815769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.815613    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:20.815769 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.815674    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj" pod "e703d5a0-297b-4e94-8686-4a490c09cd22" (UID: "e703d5a0-297b-4e94-8686-4a490c09cd22"). InnerVolumeSpecName "kube-api-access-fj5rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:20.914521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914495    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914520    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914529    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703d5a0-297b-4e94-8686-4a490c09cd22-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914539    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj5rj\" (UniqueName: \"kubernetes.io/projected/e703d5a0-297b-4e94-8686-4a490c09cd22-kube-api-access-fj5rj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914551    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914560    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703d5a0-297b-4e94-8686-4a490c09cd22-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:20.914929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:20.914569    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703d5a0-297b-4e94-8686-4a490c09cd22-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:21.698269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:21.698237    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g" event={"ID":"e703d5a0-297b-4e94-8686-4a490c09cd22","Type":"ContainerDied","Data":"f0918b8301cad02b018018c0b3488129722cc554ff5a446e0b185ebf97885573"}
Oct 30 22:53:21.698269 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:21.698261    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"
Oct 30 22:53:21.698459 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:21.698298    4410 scope.go:122] "RemoveContainer" containerID="819e48a54ba46336767962644b5937f7f0a6e5ad28e5c24751f44a62db428327"
Oct 30 22:53:21.957114 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:21.957083    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:53:21.968713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:21.968683    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.005273 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:22.005243    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-28c2g"]
Oct 30 22:53:23.019225 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:23.019193    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" path="/var/lib/kubelet/pods/e703d5a0-297b-4e94-8686-4a490c09cd22/volumes"
Oct 30 22:53:23.761445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:23.761404    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:23.938244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:23.938211    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:24.470855 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.470821    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:24.875713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.875639    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"]
Oct 30 22:53:24.876042 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876028    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy"
Oct 30 22:53:24.876093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876043    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy"
Oct 30 22:53:24.876093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876057    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fc1cc120-6383-493a-adc0-16ae80ed2215" containerName="revoke-keys"
Oct 30 22:53:24.876093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876062    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1cc120-6383-493a-adc0-16ae80ed2215" containerName="revoke-keys"
Oct 30 22:53:24.876093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876071    4410 cpu_manager.go:414] "containerMap: removing container" podUID="00c91f42-2504-4749-8cef-8b5e161a9580" containerName="revoke-keys"
Oct 30 22:53:24.876093 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876076    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c91f42-2504-4749-8cef-8b5e161a9580" containerName="revoke-keys"
Oct 30 22:53:24.876265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876147    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e703d5a0-297b-4e94-8686-4a490c09cd22" containerName="istio-proxy"
Oct 30 22:53:24.876265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876157    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1cc120-6383-493a-adc0-16ae80ed2215" containerName="revoke-keys"
Oct 30 22:53:24.876265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.876165    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c91f42-2504-4749-8cef-8b5e161a9580" containerName="revoke-keys"
Oct 30 22:53:24.880761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.880745    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:24.936696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.936672    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"]
Oct 30 22:53:24.952287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.952264    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:24.952287 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.952265    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:24.952460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.952303    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:53:24.952460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:24.952329    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:25.051329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.051295    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.051478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.051420    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqtj\" (UniqueName: \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.051478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.051453    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.152370 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.152338    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.152512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.152423    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqtj\" (UniqueName: \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.152512 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.152451    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.153053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.153032    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.154481 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.154462    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.208271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.208248    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqtj\" (UniqueName: \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\") pod \"maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.492692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.492616    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:25.863285 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:25.863260    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"]
Oct 30 22:53:25.865564 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:53:25.865532    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87887ee6_eafa_4cd2_a1e1_54f0d775c88a.slice/crio-4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a WatchSource:0}: Error finding container 4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a: Status 404 returned error can't find the container with id 4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a
Oct 30 22:53:26.730376 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:26.730332    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx" event={"ID":"87887ee6-eafa-4cd2-a1e1-54f0d775c88a","Type":"ContainerStarted","Data":"bec1c105eec009cfe8efa01433d6eb65b1b2f0235ebcd72079c3d3fac064182f"}
Oct 30 22:53:26.730824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:26.730443    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx" event={"ID":"87887ee6-eafa-4cd2-a1e1-54f0d775c88a","Type":"ContainerStarted","Data":"4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a"}
Oct 30 22:53:26.844242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:26.844196    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx" podStartSLOduration=2.844182106 podStartE2EDuration="2.844182106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:26.842573947 +0000 UTC m=+4259.931311749" watchObservedRunningTime="2026-10-30 22:53:26.844182106 +0000 UTC m=+4259.932919908"
Oct 30 22:53:27.738742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:27.738703    4410 generic.go:356] "Generic (PLEG): container finished" podID="87887ee6-eafa-4cd2-a1e1-54f0d775c88a" containerID="bec1c105eec009cfe8efa01433d6eb65b1b2f0235ebcd72079c3d3fac064182f" exitCode=0
Oct 30 22:53:27.739166 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:27.738803    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx" event={"ID":"87887ee6-eafa-4cd2-a1e1-54f0d775c88a","Type":"ContainerDied","Data":"bec1c105eec009cfe8efa01433d6eb65b1b2f0235ebcd72079c3d3fac064182f"}
Oct 30 22:53:27.842484 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:27.842452    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:28.055892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.055803    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:28.869130 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.869102    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:28.952962 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.952927    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:28.987958 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.987932    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\") pod \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") "
Oct 30 22:53:28.988106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.987980    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\" (UniqueName: \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\") pod \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") "
Oct 30 22:53:28.988106 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.988078    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\") pod \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\" (UID: \"87887ee6-eafa-4cd2-a1e1-54f0d775c88a\") "
Oct 30 22:53:28.988378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.988350    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca" pod "87887ee6-eafa-4cd2-a1e1-54f0d775c88a" (UID: "87887ee6-eafa-4cd2-a1e1-54f0d775c88a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:28.990222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.990194    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj" pod "87887ee6-eafa-4cd2-a1e1-54f0d775c88a" (UID: "87887ee6-eafa-4cd2-a1e1-54f0d775c88a"). InnerVolumeSpecName "kube-api-access-cbqtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:28.990941 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:28.990916    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca" pod "87887ee6-eafa-4cd2-a1e1-54f0d775c88a" (UID: "87887ee6-eafa-4cd2-a1e1-54f0d775c88a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:29.089696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.089670    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:29.089696 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.089694    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:29.089890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.089705    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqtj\" (UniqueName: \"kubernetes.io/projected/87887ee6-eafa-4cd2-a1e1-54f0d775c88a-kube-api-access-cbqtj\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:29.299521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.299449    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:53:29.321742 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.321718    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 22:53:29.423183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.423150    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:29.756230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.756189    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx" event={"ID":"87887ee6-eafa-4cd2-a1e1-54f0d775c88a","Type":"ContainerDied","Data":"4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a"}
Oct 30 22:53:29.756532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.756513    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4646b603efd63689e03340e065a91b5ef7f283df2923558ba0ab903d6f4d538a"
Oct 30 22:53:29.756718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.756703    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"
Oct 30 22:53:29.864204 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:29.864175    4410 scope.go:122] "RemoveContainer" containerID="ac7dd4dcbd9e8fd6d58bcf8ae1dc6d604ba09b3d835676332f759be7aa9eaefa"
Oct 30 22:53:33.882483 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:33.882456    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:36.846801 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:36.846742    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:37.069066 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:37.069031    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:37.950275 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:37.950237    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:38.760705 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:38.760674    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:43.681391 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:43.681291    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:44.319039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.319002    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerName="main" containerID="cri-o://396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc" gracePeriod=45
Oct 30 22:53:44.340347 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.340316    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main" containerID="cri-o://60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54" gracePeriod=45
Oct 30 22:53:44.586426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.586366    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:53:44.614523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.614500    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:53:44.635675 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635651    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.635844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635734    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.635844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635758    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.635844 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635837    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.636040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635869    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.636040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635953    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\" (UniqueName: \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\") pod \"0f301c5e-8f5c-4179-91f3-1f2724216584\" (UID: \"0f301c5e-8f5c-4179-91f3-1f2724216584\") "
Oct 30 22:53:44.636040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.635981    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.636040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.636024    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.636264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.636227    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.636383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.636294    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.636383 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.636338    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.638016 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.637996    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:44.638353 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.638330    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.638442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.638431    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm" pod "0f301c5e-8f5c-4179-91f3-1f2724216584" (UID: "0f301c5e-8f5c-4179-91f3-1f2724216584"). InnerVolumeSpecName "kube-api-access-f8drm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:44.736831 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.736801    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\" (UniqueName: \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.736841    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\" (UniqueName: \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.736903    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.736948    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.736992    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737041    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\") pod \"111d0a19-755a-4ba5-ad08-ae266fa61f76\" (UID: \"111d0a19-755a-4ba5-ad08-ae266fa61f76\") "
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737244    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737254    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.737294 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737277    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737464    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f301c5e-8f5c-4179-91f3-1f2724216584-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737481    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737490    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f301c5e-8f5c-4179-91f3-1f2724216584-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737499    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737508    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737519    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8drm\" (UniqueName: \"kubernetes.io/projected/0f301c5e-8f5c-4179-91f3-1f2724216584-kube-api-access-f8drm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.737667 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.737528    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.739144 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.739125    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.739448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.739431    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:44.739493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.739457    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2" pod "111d0a19-755a-4ba5-ad08-ae266fa61f76" (UID: "111d0a19-755a-4ba5-ad08-ae266fa61f76"). InnerVolumeSpecName "kube-api-access-hj6d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:44.838289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.838203    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6d2\" (UniqueName: \"kubernetes.io/projected/111d0a19-755a-4ba5-ad08-ae266fa61f76-kube-api-access-hj6d2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.838289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.838233    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111d0a19-755a-4ba5-ad08-ae266fa61f76-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.838289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.838248    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111d0a19-755a-4ba5-ad08-ae266fa61f76-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.839491 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.839463    4410 generic.go:356] "Generic (PLEG): container finished" podID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerID="396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc" exitCode=0
Oct 30 22:53:44.839633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.839516    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"
Oct 30 22:53:44.839633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.839542    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" event={"ID":"0f301c5e-8f5c-4179-91f3-1f2724216584","Type":"ContainerDied","Data":"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"}
Oct 30 22:53:44.839633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.839596    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8" event={"ID":"0f301c5e-8f5c-4179-91f3-1f2724216584","Type":"ContainerDied","Data":"61ef46ae8b18d3c283a0ca675a2c817a8047ccf4d35e6c5f0f803ee21dbe0c63"}
Oct 30 22:53:44.839633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.839619    4410 scope.go:122] "RemoveContainer" containerID="396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"
Oct 30 22:53:44.841261 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.841237    4410 generic.go:356] "Generic (PLEG): container finished" podID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerID="60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54" exitCode=0
Oct 30 22:53:44.841374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.841320    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" event={"ID":"111d0a19-755a-4ba5-ad08-ae266fa61f76","Type":"ContainerDied","Data":"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"}
Oct 30 22:53:44.841374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.841351    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr" event={"ID":"111d0a19-755a-4ba5-ad08-ae266fa61f76","Type":"ContainerDied","Data":"25daeffd08741146af21e6eabf1d13d4f8dbbd5835d600f71067bd262716df65"}
Oct 30 22:53:44.841467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.841419    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"
Oct 30 22:53:44.851796 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.851751    4410 scope.go:122] "RemoveContainer" containerID="396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"
Oct 30 22:53:44.852074 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:53:44.852046    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc\": container with ID starting with 396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc not found: ID does not exist" containerID="396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"
Oct 30 22:53:44.852198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.852085    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc"} err="failed to get container status \"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc\": rpc error: code = NotFound desc = could not find container \"396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc\": container with ID starting with 396f2b97920ccfa20e4c3f22651462a330e459c8eaffe225030f00b2cb70ebdc not found: ID does not exist"
Oct 30 22:53:44.852198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.852108    4410 scope.go:122] "RemoveContainer" containerID="60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"
Oct 30 22:53:44.862021 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.861998    4410 scope.go:122] "RemoveContainer" containerID="60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"
Oct 30 22:53:44.862279 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:53:44.862258    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54\": container with ID starting with 60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54 not found: ID does not exist" containerID="60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"
Oct 30 22:53:44.862356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:44.862285    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54"} err="failed to get container status \"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54\": rpc error: code = NotFound desc = could not find container \"60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54\": container with ID starting with 60da0662aed0c8d15e09a800cbdbed9eb11d25de72c02a0987fbbdcbdfa0dd54 not found: ID does not exist"
Oct 30 22:53:45.001791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:45.001747    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:53:45.005417 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:45.005391    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-w2-90e0d5-kserve-8f8678fc6-6gkl8"]
Oct 30 22:53:45.018415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:45.018388    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" path="/var/lib/kubelet/pods/0f301c5e-8f5c-4179-91f3-1f2724216584/volumes"
Oct 30 22:53:45.112765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:45.112708    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 22:53:45.114736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:45.114715    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/simulator-w2-90e0d5-kserve-746b48b6f6-cjsvr"]
Oct 30 22:53:47.017940 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:47.017909    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" path="/var/lib/kubelet/pods/111d0a19-755a-4ba5-ad08-ae266fa61f76/volumes"
Oct 30 22:53:47.944340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:47.944312    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:49.454558 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:49.454519    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:49.721798 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:49.721696    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:49.866159 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:49.866125    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:49.925169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:49.925142    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"]
Oct 30 22:53:49.974492 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:49.974417    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b8717e-5fs2g"]
Oct 30 22:53:50.352212 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:50.352131    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:50.520985 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:53:50.520952    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:53:51.018542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:51.018508    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835dde92-eb75-4747-8618-a50a6058406c" path="/var/lib/kubelet/pods/835dde92-eb75-4747-8618-a50a6058406c/volumes"
Oct 30 22:53:55.012375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:55.012335    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:56.467791 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:56.467747    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:56.619400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:56.619361    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:56.753080 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:56.753005    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:53:56.753234 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:56.753215    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" containerID="cri-o://1f7c34992361300f6a826f2316f62cf2147d8de4557ff9c49ddf5e92aa30bf19" gracePeriod=30
Oct 30 22:53:56.776252 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:56.776227    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:58.142408 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:58.142368    4410 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:53:58.142803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:58.142443    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:53:59.858424 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:53:59.858393    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:01.939359 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:01.939323    4410 generic.go:356] "Generic (PLEG): container finished" podID="90409607-8a45-4049-94ac-49e394950d33" containerID="1f7c34992361300f6a826f2316f62cf2147d8de4557ff9c49ddf5e92aa30bf19" exitCode=0
Oct 30 22:54:01.939697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:01.939405    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" event={"ID":"90409607-8a45-4049-94ac-49e394950d33","Type":"ContainerDied","Data":"1f7c34992361300f6a826f2316f62cf2147d8de4557ff9c49ddf5e92aa30bf19"}
Oct 30 22:54:02.008316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.008294    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:54:02.097719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097657    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.097719 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097702    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.097920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097744    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.097920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097764    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.097920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097816    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.097920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097862    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097942    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098004    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.097940    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098072    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098120    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\") pod \"90409607-8a45-4049-94ac-49e394950d33\" (UID: \"90409607-8a45-4049-94ac-49e394950d33\") "
Oct 30 22:54:02.098158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098140    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098387    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098409    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098487    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90409607-8a45-4049-94ac-49e394950d33-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098499    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098508    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098517    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.098543 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.098529    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.100045 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.100026    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:02.100164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.100047    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:02.100457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.100431    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:54:02.100531 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.100451    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2" pod "90409607-8a45-4049-94ac-49e394950d33" (UID: "90409607-8a45-4049-94ac-49e394950d33"). InnerVolumeSpecName "kube-api-access-2sdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:02.199586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.199553    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdb2\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-kube-api-access-2sdb2\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.199586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.199583    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90409607-8a45-4049-94ac-49e394950d33-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.199586 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.199593    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90409607-8a45-4049-94ac-49e394950d33-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.199827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.199606    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90409607-8a45-4049-94ac-49e394950d33-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:02.946098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.946067    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"
Oct 30 22:54:02.946552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.946098    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72" event={"ID":"90409607-8a45-4049-94ac-49e394950d33","Type":"ContainerDied","Data":"29e2eb71647c765221442d774eb56c37ebae97c84e5c28c75033e8353e84f7b1"}
Oct 30 22:54:02.946552 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:02.946156    4410 scope.go:122] "RemoveContainer" containerID="1f7c34992361300f6a826f2316f62cf2147d8de4557ff9c49ddf5e92aa30bf19"
Oct 30 22:54:03.167960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:03.167923    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:54:03.217344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:03.217262    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-fxs72"]
Oct 30 22:54:04.322077 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322046    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"]
Oct 30 22:54:04.322445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322432    4410 cpu_manager.go:414] "containerMap: removing container" podUID="87887ee6-eafa-4cd2-a1e1-54f0d775c88a" containerName="revoke-keys"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322446    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87887ee6-eafa-4cd2-a1e1-54f0d775c88a" containerName="revoke-keys"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322461    4410 cpu_manager.go:414] "containerMap: removing container" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322467    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322481    4410 cpu_manager.go:414] "containerMap: removing container" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerName="main"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322485    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerName="main"
Oct 30 22:54:04.322493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322495    4410 cpu_manager.go:414] "containerMap: removing container" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy"
Oct 30 22:54:04.322687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322499    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy"
Oct 30 22:54:04.322687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322566    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="87887ee6-eafa-4cd2-a1e1-54f0d775c88a" containerName="revoke-keys"
Oct 30 22:54:04.322687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322574    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="111d0a19-755a-4ba5-ad08-ae266fa61f76" containerName="main"
Oct 30 22:54:04.322687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322582    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f301c5e-8f5c-4179-91f3-1f2724216584" containerName="main"
Oct 30 22:54:04.322687 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.322590    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="90409607-8a45-4049-94ac-49e394950d33" containerName="istio-proxy"
Oct 30 22:54:04.327013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.326993    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.372013 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.371984    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"]
Oct 30 22:54:04.375496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.375477    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.378864 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.378843    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:04.419126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419095    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:04.419271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419156    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:04.419271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419158    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:54:04.419711 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419690    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.419753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419733    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.419861 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.419843    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfptz\" (UniqueName: \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.421438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.421418    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"]
Oct 30 22:54:04.435972 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.435950    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"]
Oct 30 22:54:04.474164 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.474133    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:04.520651 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520616    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.520826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520687    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.520826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520738    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.520826 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520767    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.520999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520830    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfptz\" (UniqueName: \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.520999 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.520892    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw9nz\" (UniqueName: \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.521467 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.521445    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.523373 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.523335    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.578710 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.578657    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfptz\" (UniqueName: \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\") pod \"maas-api-revoke-sub-3f0b9466-mvz2m\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.622181 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.622159    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.622279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.622205    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9nz\" (UniqueName: \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.622279 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.622261    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.623009 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.622986    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.624256 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.624238    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.641248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.641228    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:04.679685 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.679623    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9nz\" (UniqueName: \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\") pod \"maas-api-revoke-sub-dc34ff0e-nzmnv\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:04.688563 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:04.688533    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:05.009575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.009537    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"]
Oct 30 22:54:05.018720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.018689    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90409607-8a45-4049-94ac-49e394950d33" path="/var/lib/kubelet/pods/90409607-8a45-4049-94ac-49e394950d33/volumes"
Oct 30 22:54:05.051536 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:54:05.051507    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8207005e_4af3_47cf_be89_5dd18d231650.slice/crio-a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c WatchSource:0}: Error finding container a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c: Status 404 returned error can't find the container with id a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c
Oct 30 22:54:05.065818 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.063074    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"]
Oct 30 22:54:05.067718 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:54:05.067687    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a07865_7bb7_41f8_b650_69095e8eca0c.slice/crio-8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564 WatchSource:0}: Error finding container 8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564: Status 404 returned error can't find the container with id 8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564
Oct 30 22:54:05.160063 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.160036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:05.569879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.569767    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:05.763193 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.763153    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:05.951797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.951685    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:05.965741 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.965715    4410 generic.go:356] "Generic (PLEG): container finished" podID="8207005e-4af3-47cf-be89-5dd18d231650" containerID="7bd8c8ff186f2f6a8ded87fdbe1b99d9c50c0d82812dcd6fe5a7a014893dc6b9" exitCode=0
Oct 30 22:54:05.965909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.965815    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m" event={"ID":"8207005e-4af3-47cf-be89-5dd18d231650","Type":"ContainerDied","Data":"7bd8c8ff186f2f6a8ded87fdbe1b99d9c50c0d82812dcd6fe5a7a014893dc6b9"}
Oct 30 22:54:05.965909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.965877    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m" event={"ID":"8207005e-4af3-47cf-be89-5dd18d231650","Type":"ContainerStarted","Data":"a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c"}
Oct 30 22:54:05.967966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.967942    4410 generic.go:356] "Generic (PLEG): container finished" podID="46a07865-7bb7-41f8-b650-69095e8eca0c" containerID="18228513757dea64b4af3349f8de4b90813a5df5121b6098f114c3d292cec94a" exitCode=0
Oct 30 22:54:05.968069 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.968040    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv" event={"ID":"46a07865-7bb7-41f8-b650-69095e8eca0c","Type":"ContainerDied","Data":"18228513757dea64b4af3349f8de4b90813a5df5121b6098f114c3d292cec94a"}
Oct 30 22:54:05.968125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:05.968083    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv" event={"ID":"46a07865-7bb7-41f8-b650-69095e8eca0c","Type":"ContainerStarted","Data":"8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564"}
Oct 30 22:54:06.148199 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:06.148160    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:06.335480 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:06.335387    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:07.114670 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.114641    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:07.123070 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.123048    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:07.142438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.142413    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:07.249241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249156    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\") pod \"8207005e-4af3-47cf-be89-5dd18d231650\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") "
Oct 30 22:54:07.249241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249209    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\") pod \"8207005e-4af3-47cf-be89-5dd18d231650\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") "
Oct 30 22:54:07.249489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249254    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\" (UniqueName: \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\") pod \"8207005e-4af3-47cf-be89-5dd18d231650\" (UID: \"8207005e-4af3-47cf-be89-5dd18d231650\") "
Oct 30 22:54:07.249489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249346    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\") pod \"46a07865-7bb7-41f8-b650-69095e8eca0c\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") "
Oct 30 22:54:07.249489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249417    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\" (UniqueName: \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\") pod \"46a07865-7bb7-41f8-b650-69095e8eca0c\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") "
Oct 30 22:54:07.249664 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249491    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\") pod \"46a07865-7bb7-41f8-b650-69095e8eca0c\" (UID: \"46a07865-7bb7-41f8-b650-69095e8eca0c\") "
Oct 30 22:54:07.249724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249693    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca" pod "46a07865-7bb7-41f8-b650-69095e8eca0c" (UID: "46a07865-7bb7-41f8-b650-69095e8eca0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:07.249954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249926    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca" pod "8207005e-4af3-47cf-be89-5dd18d231650" (UID: "8207005e-4af3-47cf-be89-5dd18d231650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:07.250092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.249954    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.251536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.251504    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz" pod "8207005e-4af3-47cf-be89-5dd18d231650" (UID: "8207005e-4af3-47cf-be89-5dd18d231650"). InnerVolumeSpecName "kube-api-access-zfptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:07.251649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.251534    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca" pod "46a07865-7bb7-41f8-b650-69095e8eca0c" (UID: "46a07865-7bb7-41f8-b650-69095e8eca0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:07.251720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.251655    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca" pod "8207005e-4af3-47cf-be89-5dd18d231650" (UID: "8207005e-4af3-47cf-be89-5dd18d231650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:07.252202 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.252185    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz" pod "46a07865-7bb7-41f8-b650-69095e8eca0c" (UID: "46a07865-7bb7-41f8-b650-69095e8eca0c"). InnerVolumeSpecName "kube-api-access-cw9nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:07.272765 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.272736    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:07.350797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.350743    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw9nz\" (UniqueName: \"kubernetes.io/projected/46a07865-7bb7-41f8-b650-69095e8eca0c-kube-api-access-cw9nz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.350797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.350788    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a07865-7bb7-41f8-b650-69095e8eca0c-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.350797 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.350799    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.351061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.350808    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207005e-4af3-47cf-be89-5dd18d231650-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.351061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.350818    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfptz\" (UniqueName: \"kubernetes.io/projected/8207005e-4af3-47cf-be89-5dd18d231650-kube-api-access-zfptz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:07.420954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.420921    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:07.980476 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.980441    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"
Oct 30 22:54:07.980648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.980446    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m" event={"ID":"8207005e-4af3-47cf-be89-5dd18d231650","Type":"ContainerDied","Data":"a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c"}
Oct 30 22:54:07.980648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.980588    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9294825806ec60b890961eb8b26f5d814a93111f5708c524211823af625290c"
Oct 30 22:54:07.981929 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.981906    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"
Oct 30 22:54:07.982058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.981909    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv" event={"ID":"46a07865-7bb7-41f8-b650-69095e8eca0c","Type":"ContainerDied","Data":"8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564"}
Oct 30 22:54:07.982058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:07.982015    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de9d55f7beb558a2e8b76a0b3413a0659df0724c490eb2ecb445c34a5926564"
Oct 30 22:54:08.347600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:08.347516    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:10.065963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:10.065923    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:14.179334 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:14.179298    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:14.335226 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:14.335195    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:14.963527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:14.963471    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:18.984392 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:18.984355    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:19.186072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:19.186036    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:19.703198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:19.703167    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:19.884301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:19.884264    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:20.521513 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:54:20.521478    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:54:24.757372 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:24.757331    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:25.789098 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:25.789064    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:25.978004 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:25.977964    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:29.553637 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:29.553598    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:29.904585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:29.904549    4410 scope.go:122] "RemoveContainer" containerID="bf95ee16a9a51fdd7438cadf89c2f83cf09dfe742f14bdbfad2fe62a7b61f44b"
Oct 30 22:54:31.936489 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:31.936448    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:32.092517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.092481    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:32.468183 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468152    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"]
Oct 30 22:54:32.468513 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468500    4410 cpu_manager.go:414] "containerMap: removing container" podUID="46a07865-7bb7-41f8-b650-69095e8eca0c" containerName="revoke-keys"
Oct 30 22:54:32.468566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468514    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a07865-7bb7-41f8-b650-69095e8eca0c" containerName="revoke-keys"
Oct 30 22:54:32.468566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468545    4410 cpu_manager.go:414] "containerMap: removing container" podUID="8207005e-4af3-47cf-be89-5dd18d231650" containerName="revoke-keys"
Oct 30 22:54:32.468566 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468550    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8207005e-4af3-47cf-be89-5dd18d231650" containerName="revoke-keys"
Oct 30 22:54:32.468682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468617    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="8207005e-4af3-47cf-be89-5dd18d231650" containerName="revoke-keys"
Oct 30 22:54:32.468682 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.468629    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a07865-7bb7-41f8-b650-69095e8eca0c" containerName="revoke-keys"
Oct 30 22:54:32.473455 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.473439    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.528683 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.528652    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"]
Oct 30 22:54:32.571966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.571935    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.572094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.572000    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.572161 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.572138    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnm4\" (UniqueName: \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.574926 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.574909    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:32.574989 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.574945    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:32.624401 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.624377    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:32.624521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.624439    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:54:32.673448 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.673425    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnm4\" (UniqueName: \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.673560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.673456    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.673560 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.673489    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.674095 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.674076    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.675786 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.675755    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.729760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.729702    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnm4\" (UniqueName: \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\") pod \"maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:32.787157 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:32.787134    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:33.162538 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:33.162514    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"]
Oct 30 22:54:33.164437 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:54:33.164404    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b39cfd_7ff0_4978_85bb_99c741d0436c.slice/crio-2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3 WatchSource:0}: Error finding container 2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3: Status 404 returned error can't find the container with id 2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3
Oct 30 22:54:34.125901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:34.125874    4410 generic.go:356] "Generic (PLEG): container finished" podID="d1b39cfd-7ff0-4978-85bb-99c741d0436c" containerID="64d43d807ae14dda48ccd8ce1b65e1ce42e4e1dc594ff6417083516353805d47" exitCode=0
Oct 30 22:54:34.126036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:34.125952    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll" event={"ID":"d1b39cfd-7ff0-4978-85bb-99c741d0436c","Type":"ContainerDied","Data":"64d43d807ae14dda48ccd8ce1b65e1ce42e4e1dc594ff6417083516353805d47"}
Oct 30 22:54:34.126036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:34.126000    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll" event={"ID":"d1b39cfd-7ff0-4978-85bb-99c741d0436c","Type":"ContainerStarted","Data":"2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3"}
Oct 30 22:54:34.474925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:34.474894    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:35.263412 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.263388    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:35.401278 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.401253    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\") pod \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") "
Oct 30 22:54:35.401431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.401334    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\" (UniqueName: \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\") pod \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") "
Oct 30 22:54:35.401431 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.401364    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\") pod \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\" (UID: \"d1b39cfd-7ff0-4978-85bb-99c741d0436c\") "
Oct 30 22:54:35.401581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.401559    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca" pod "d1b39cfd-7ff0-4978-85bb-99c741d0436c" (UID: "d1b39cfd-7ff0-4978-85bb-99c741d0436c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:35.403474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.403447    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4" pod "d1b39cfd-7ff0-4978-85bb-99c741d0436c" (UID: "d1b39cfd-7ff0-4978-85bb-99c741d0436c"). InnerVolumeSpecName "kube-api-access-qvnm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:35.403573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.403487    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca" pod "d1b39cfd-7ff0-4978-85bb-99c741d0436c" (UID: "d1b39cfd-7ff0-4978-85bb-99c741d0436c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:35.502366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.502329    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:35.502366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.502362    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b39cfd-7ff0-4978-85bb-99c741d0436c-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:35.502752 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:35.502378    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnm4\" (UniqueName: \"kubernetes.io/projected/d1b39cfd-7ff0-4978-85bb-99c741d0436c-kube-api-access-qvnm4\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:36.138158 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:36.138127    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"
Oct 30 22:54:36.138339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:36.138163    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll" event={"ID":"d1b39cfd-7ff0-4978-85bb-99c741d0436c","Type":"ContainerDied","Data":"2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3"}
Oct 30 22:54:36.138339 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:36.138209    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2318aefc9f06aee4c6120b1a2a245e5b1e7e1cfb8b52a898d1219d75701342c3"
Oct 30 22:54:38.368209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:38.368169    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.123548 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:39.123508    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:40.822224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:40.822189    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.104248 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:41.104167    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.253453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:41.253420    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.304901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:41.304869    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"]
Oct 30 22:54:41.354015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:41.353989    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90e0d5-g4dfx"]
Oct 30 22:54:41.722498 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:41.722461    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:43.018155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:43.018121    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87887ee6-eafa-4cd2-a1e1-54f0d775c88a" path="/var/lib/kubelet/pods/87887ee6-eafa-4cd2-a1e1-54f0d775c88a/volumes"
Oct 30 22:54:44.389508 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:44.389468    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:44.591117 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:44.591076    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:46.495367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:46.495333    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:47.908388 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:47.908351    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:48.082400 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:48.082364    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:48.259264 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:48.259177    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:50.520922 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:54:50.520884    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:54:51.343523 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:51.343482    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:56.256547 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:54:56.256517    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:01.040447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:01.040412    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:05.290837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:05.290799    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:06.828154 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:06.828117    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:07.068142 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:07.068106    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:07.232642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:07.232609    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:07.319561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:07.319529    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"]
Oct 30 22:55:07.368449 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:07.368425    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6924e5-mmgll"]
Oct 30 22:55:09.017977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:09.017946    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b39cfd-7ff0-4978-85bb-99c741d0436c" path="/var/lib/kubelet/pods/d1b39cfd-7ff0-4978-85bb-99c741d0436c/volumes"
Oct 30 22:55:12.264970 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:12.264882    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:17.157565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:17.157522    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:17.748614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:17.748581    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:20.521099 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:55:20.521050    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:55:22.155587 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:22.155550    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:23.856633 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:23.856599    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:24.051925 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:24.051888    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:24.162442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:24.162413    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:55:24.162677 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:24.162656    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" containerID="cri-o://9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2" gracePeriod=30
Oct 30 22:55:24.268514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:24.268481    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:25.303068 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:25.303033    4410 patch_prober.go:29] interesting pod/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:55:25.303478 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:25.303115    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:55:26.961932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:26.961899    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:29.420148 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.420121    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:55:29.432735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.432673    4410 generic.go:356] "Generic (PLEG): container finished" podID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerID="9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2" exitCode=0
Oct 30 22:55:29.432735 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.432721    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"
Oct 30 22:55:29.432923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.432762    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerDied","Data":"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"}
Oct 30 22:55:29.432923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.432825    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km" event={"ID":"e2820b03-a73f-4f04-94c5-40adab80bbef","Type":"ContainerDied","Data":"9ad378bd9191a13d5f3840eca5316215bc9ee86a197135ed2b71f69baf6d8e43"}
Oct 30 22:55:29.432923 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.432853    4410 scope.go:122] "RemoveContainer" containerID="9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"
Oct 30 22:55:29.441692 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.441668    4410 scope.go:122] "RemoveContainer" containerID="468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"
Oct 30 22:55:29.449963 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.449945    4410 scope.go:122] "RemoveContainer" containerID="9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"
Oct 30 22:55:29.450204 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:55:29.450184    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2\": container with ID starting with 9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2 not found: ID does not exist" containerID="9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"
Oct 30 22:55:29.450271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.450215    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2"} err="failed to get container status \"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2\": rpc error: code = NotFound desc = could not find container \"9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2\": container with ID starting with 9eac591d6f022353e38b792a7b7fda12e4829beae31d2ce2289d2a54c36c15e2 not found: ID does not exist"
Oct 30 22:55:29.450271 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.450233    4410 scope.go:122] "RemoveContainer" containerID="468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"
Oct 30 22:55:29.450520 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:55:29.450502    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532\": container with ID starting with 468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532 not found: ID does not exist" containerID="468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"
Oct 30 22:55:29.450575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.450525    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532"} err="failed to get container status \"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532\": rpc error: code = NotFound desc = could not find container \"468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532\": container with ID starting with 468eb7ce70cb308e0a852098e5362e326d4d6a18f7f07fddc0a3d3b30bea3532 not found: ID does not exist"
Oct 30 22:55:29.492058 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492038    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492082    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492110    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492155 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492136    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492158    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492178    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492202    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492241    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492344 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492266    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\") pod \"e2820b03-a73f-4f04-94c5-40adab80bbef\" (UID: \"e2820b03-a73f-4f04-94c5-40adab80bbef\") "
Oct 30 22:55:29.492632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492468    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:29.492700 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492659    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.492760 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492694    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:29.492846 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492799    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:29.492920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492885    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:29.492920 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.492626    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:29.494237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.494212    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:55:29.494237 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.494222    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "kube-api-access-ntkv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:29.494352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.494243    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:29.494352 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.494319    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token" pod "e2820b03-a73f-4f04-94c5-40adab80bbef" (UID: "e2820b03-a73f-4f04-94c5-40adab80bbef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:29.593222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593198    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593222    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593232    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593241    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2820b03-a73f-4f04-94c5-40adab80bbef-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593249    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593259    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593267    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2820b03-a73f-4f04-94c5-40adab80bbef-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.593367 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.593276    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkv7\" (UniqueName: \"kubernetes.io/projected/e2820b03-a73f-4f04-94c5-40adab80bbef-kube-api-access-ntkv7\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.852784 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.852748    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:55:29.856469 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.856442    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6924e5-openshift-default-648b667bb9-q95km"]
Oct 30 22:55:29.940954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.940926    4410 scope.go:122] "RemoveContainer" containerID="5c87bf66816a9f45f92ad73de05ec72139703f729508be83cf8f2b4974ce9524"
Oct 30 22:55:29.949732 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:29.949715    4410 scope.go:122] "RemoveContainer" containerID="3d8a0dfb3119f4f7d9d5d636a54879876a78c5fbd7c79676c17442d0a1561fd8"
Oct 30 22:55:31.018983 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:31.018950    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" path="/var/lib/kubelet/pods/e2820b03-a73f-4f04-94c5-40adab80bbef/volumes"
Oct 30 22:55:31.756790 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:31.756745    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.007331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007294    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"]
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007667    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007679    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007702    4410 cpu_manager.go:414] "containerMap: removing container" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007707    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007719    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d1b39cfd-7ff0-4978-85bb-99c741d0436c" containerName="revoke-keys"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007725    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b39cfd-7ff0-4978-85bb-99c741d0436c" containerName="revoke-keys"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007808    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:55:35.007859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.007817    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b39cfd-7ff0-4978-85bb-99c741d0436c" containerName="revoke-keys"
Oct 30 22:55:35.012524 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.012500    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.068241 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.068207    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"]
Oct 30 22:55:35.114712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.114681    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:35.114712 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.114703    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:35.114921 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.114834    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:35.114965 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.114937    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:55:35.142643 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.142623    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.142799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.142671    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.142799 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.142790    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nr5\" (UniqueName: \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.244060 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.244039    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.244169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.244092    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.244169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.244132    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nr5\" (UniqueName: \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.244726 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.244701    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.246258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.246238    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.300787 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.300713    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nr5\" (UniqueName: \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\") pod \"maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.325473 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.325453    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:35.693452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:35.693417    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"]
Oct 30 22:55:35.696037 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:55:35.696010    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e2c50c_5977_48ba_8a3c_850ee844e25b.slice/crio-5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d WatchSource:0}: Error finding container 5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d: Status 404 returned error can't find the container with id 5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d
Oct 30 22:55:36.470511 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:36.470473    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z" event={"ID":"f3e2c50c-5977-48ba-8a3c-850ee844e25b","Type":"ContainerStarted","Data":"585e7bca930793f5e97db5c9f8c269bf832ad6a2a30574c2f5b600d86cb7bcd7"}
Oct 30 22:55:36.470918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:36.470536    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z" event={"ID":"f3e2c50c-5977-48ba-8a3c-850ee844e25b","Type":"ContainerStarted","Data":"5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d"}
Oct 30 22:55:36.583699 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:36.583657    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z" podStartSLOduration=2.5836421 podStartE2EDuration="2.5836421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:36.583560585 +0000 UTC m=+4389.672298388" watchObservedRunningTime="2026-10-30 22:55:36.5836421 +0000 UTC m=+4389.672379903"
Oct 30 22:55:36.880642 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:36.880562    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:37.477571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:37.477539    4410 generic.go:356] "Generic (PLEG): container finished" podID="f3e2c50c-5977-48ba-8a3c-850ee844e25b" containerID="585e7bca930793f5e97db5c9f8c269bf832ad6a2a30574c2f5b600d86cb7bcd7" exitCode=0
Oct 30 22:55:37.478003 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:37.477633    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z" event={"ID":"f3e2c50c-5977-48ba-8a3c-850ee844e25b","Type":"ContainerDied","Data":"585e7bca930793f5e97db5c9f8c269bf832ad6a2a30574c2f5b600d86cb7bcd7"}
Oct 30 22:55:38.614125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.614094    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:38.624385 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.624357    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:38.775689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.775614    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\") pod \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") "
Oct 30 22:55:38.775689 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.775665    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\" (UniqueName: \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\") pod \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") "
Oct 30 22:55:38.775901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.775767    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\") pod \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\" (UID: \"f3e2c50c-5977-48ba-8a3c-850ee844e25b\") "
Oct 30 22:55:38.776046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.776018    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca" pod "f3e2c50c-5977-48ba-8a3c-850ee844e25b" (UID: "f3e2c50c-5977-48ba-8a3c-850ee844e25b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:38.776153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.776134    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:38.777829 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.777801    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5" pod "f3e2c50c-5977-48ba-8a3c-850ee844e25b" (UID: "f3e2c50c-5977-48ba-8a3c-850ee844e25b"). InnerVolumeSpecName "kube-api-access-f5nr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:38.777932 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.777885    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca" pod "f3e2c50c-5977-48ba-8a3c-850ee844e25b" (UID: "f3e2c50c-5977-48ba-8a3c-850ee844e25b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:38.876863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.876837    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e2c50c-5977-48ba-8a3c-850ee844e25b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:38.876863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:38.876863    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nr5\" (UniqueName: \"kubernetes.io/projected/f3e2c50c-5977-48ba-8a3c-850ee844e25b-kube-api-access-f5nr5\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:39.488612 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:39.488580    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"
Oct 30 22:55:39.488768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:39.488631    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z" event={"ID":"f3e2c50c-5977-48ba-8a3c-850ee844e25b","Type":"ContainerDied","Data":"5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d"}
Oct 30 22:55:39.488768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:39.488675    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f126ac6dba506b2481cce73872f5f000622b528f7de167551bcdde136a3ab2d"
Oct 30 22:55:42.040406 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:42.040367    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:46.307355 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:46.307313    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:46.823219 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:46.823186    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:47.033030 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:47.032993    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:48.151222 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:48.151190    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:50.520976 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:55:50.520929    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:55:52.955619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:52.955586    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:53.154198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:53.154158    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:53.414427 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:53.414398    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:53.610832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:53.610797    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"]
Oct 30 22:55:53.659857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:53.659821    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3faf84-lf29z"]
Oct 30 22:55:54.013837 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:54.013801    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:55.018322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:55.018286    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e2c50c-5977-48ba-8a3c-850ee844e25b" path="/var/lib/kubelet/pods/f3e2c50c-5977-48ba-8a3c-850ee844e25b/volumes"
Oct 30 22:55:58.166297 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:55:58.166255    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:00.152243 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:00.152208    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:00.365613 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:00.365574    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:00.526356 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:00.526271    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:01.279056 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.279026    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"]
Oct 30 22:56:01.279436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.279378    4410 cpu_manager.go:414] "containerMap: removing container" podUID="f3e2c50c-5977-48ba-8a3c-850ee844e25b" containerName="revoke-keys"
Oct 30 22:56:01.279436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.279389    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e2c50c-5977-48ba-8a3c-850ee844e25b" containerName="revoke-keys"
Oct 30 22:56:01.279520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.279466    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2820b03-a73f-4f04-94c5-40adab80bbef" containerName="istio-proxy"
Oct 30 22:56:01.279520 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.279478    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e2c50c-5977-48ba-8a3c-850ee844e25b" containerName="revoke-keys"
Oct 30 22:56:01.282648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.282631    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.337374 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.337327    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:01.362326 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.362297    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj85w\" (UniqueName: \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.362460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.362353    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.362460 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.362371    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.386506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.386485    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:01.386506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.386500    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:01.386661 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.386640    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:56:01.388575 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.388555    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"]
Oct 30 22:56:01.462909 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.462884    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj85w\" (UniqueName: \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.463039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.462936    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.463039 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.462992    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.463494 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.463476    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.465259 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.465240    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.519593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.519567    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj85w\" (UniqueName: \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.595210 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.595141    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:01.966182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:01.966160    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"]
Oct 30 22:56:01.969860 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:56:01.969828    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27a1957_8279_456e_b6ca_5244e1fa092b.slice/crio-b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6 WatchSource:0}: Error finding container b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6: Status 404 returned error can't find the container with id b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6
Oct 30 22:56:02.615555 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:02.615514    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt" event={"ID":"d27a1957-8279-456e-b6ca-5244e1fa092b","Type":"ContainerStarted","Data":"b3825bf29ee06e0ebc21f89c5cfa2562fb7c46f836328272512b90662cff684a"}
Oct 30 22:56:02.616040 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:02.615602    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt" event={"ID":"d27a1957-8279-456e-b6ca-5244e1fa092b","Type":"ContainerStarted","Data":"b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6"}
Oct 30 22:56:02.729713 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:02.729672    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt" podStartSLOduration=1.729659517 podStartE2EDuration="1.729659517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:02.727451845 +0000 UTC m=+4415.816189651" watchObservedRunningTime="2026-10-30 22:56:02.729659517 +0000 UTC m=+4415.818397320"
Oct 30 22:56:03.084720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:03.084678    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:03.623591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:03.623555    4410 generic.go:356] "Generic (PLEG): container finished" podID="d27a1957-8279-456e-b6ca-5244e1fa092b" containerID="b3825bf29ee06e0ebc21f89c5cfa2562fb7c46f836328272512b90662cff684a" exitCode=0
Oct 30 22:56:03.624015 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:03.623632    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt" event={"ID":"d27a1957-8279-456e-b6ca-5244e1fa092b","Type":"ContainerDied","Data":"b3825bf29ee06e0ebc21f89c5cfa2562fb7c46f836328272512b90662cff684a"}
Oct 30 22:56:04.750211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.750186    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:04.790756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.790728    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\") pod \"d27a1957-8279-456e-b6ca-5244e1fa092b\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") "
Oct 30 22:56:04.790879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.790827    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\" (UniqueName: \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\") pod \"d27a1957-8279-456e-b6ca-5244e1fa092b\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") "
Oct 30 22:56:04.790879 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.790860    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\") pod \"d27a1957-8279-456e-b6ca-5244e1fa092b\" (UID: \"d27a1957-8279-456e-b6ca-5244e1fa092b\") "
Oct 30 22:56:04.791105 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.791081    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca" pod "d27a1957-8279-456e-b6ca-5244e1fa092b" (UID: "d27a1957-8279-456e-b6ca-5244e1fa092b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:04.792832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.792753    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca" pod "d27a1957-8279-456e-b6ca-5244e1fa092b" (UID: "d27a1957-8279-456e-b6ca-5244e1fa092b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:04.792832 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.792821    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w" pod "d27a1957-8279-456e-b6ca-5244e1fa092b" (UID: "d27a1957-8279-456e-b6ca-5244e1fa092b"). InnerVolumeSpecName "kube-api-access-cj85w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:04.892024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.891997    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.892024 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.892021    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj85w\" (UniqueName: \"kubernetes.io/projected/d27a1957-8279-456e-b6ca-5244e1fa092b-kube-api-access-cj85w\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.892249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:04.892031    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27a1957-8279-456e-b6ca-5244e1fa092b-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.635817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:05.635783    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"
Oct 30 22:56:05.636000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:05.635789    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt" event={"ID":"d27a1957-8279-456e-b6ca-5244e1fa092b","Type":"ContainerDied","Data":"b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6"}
Oct 30 22:56:05.636000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:05.635895    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b191f4596cdb67c15981c9613e3f7e53d54f9a80697144c8738af651522458d6"
Oct 30 22:56:07.609280 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:07.609251    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:08.041648 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:08.041605    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:12.282931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:12.282893    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:12.647442 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:12.647411    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:56:12.842817 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:12.842767    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.021329 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:13.021240    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:14.169747 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:14.169713    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:16.948727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:16.948686    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:17.110852 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.110814    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:17.286600 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.286517    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:17.434341 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.434306    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"]
Oct 30 22:56:17.483044 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.482991    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89cf28-8gggt"]
Oct 30 22:56:17.648506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.648469    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing" containerID="cri-o://16ed0ab553c68e09dbf6f40e2ce7b88cd435af0476721419600f6a4ef9821309" gracePeriod=25
Oct 30 22:56:17.850475 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:17.850442    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:19.017891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:19.017860    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27a1957-8279-456e-b6ca-5244e1fa092b" path="/var/lib/kubelet/pods/d27a1957-8279-456e-b6ca-5244e1fa092b/volumes"
Oct 30 22:56:19.224892 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:19.224860    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:19.781375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:19.781336    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:20.521335 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:56:20.521299    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:56:23.900915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:23.900874    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:24.120977 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:24.120941    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:24.282991 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:24.282900    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:24.451304 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:24.451268    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:29.503497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:29.503463    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:29.781720 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:29.781636    4410 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:29.980990 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:29.980959    4410 scope.go:122] "RemoveContainer" containerID="eb3b5c4df498e4915bd7cb411c442f9be0b8c3628054410ef06e71722513eac3"
Oct 30 22:56:29.992540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:29.992515    4410 scope.go:122] "RemoveContainer" containerID="9f36cfb7abd1d5f8431f9ae3d239833ae391771df840ed68aab458eb5c55ce0a"
Oct 30 22:56:30.000811 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:30.000791    4410 scope.go:122] "RemoveContainer" containerID="ca727aa51f9e3c7e5cccb50e91de2333edf5e466b5210d1f09e3331f6a1a210d"
Oct 30 22:56:30.008988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:30.008974    4410 scope.go:122] "RemoveContainer" containerID="5e37501505c8c5550712e8866fe756b1952f44e8b64e2c1e3c171fbc418b3015"
Oct 30 22:56:30.016270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:30.016247    4410 scope.go:122] "RemoveContainer" containerID="ebd39341ed0704ca00b91064ae79644928abe79ab417183bae82029cb01a48cb"
Oct 30 22:56:34.457462 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:34.457427    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:36.933540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:36.933507    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:37.086872 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.086824    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:37.657814 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.657747    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:37.816581 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.816550    4410 generic.go:356] "Generic (PLEG): container finished" podID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerID="16ed0ab553c68e09dbf6f40e2ce7b88cd435af0476721419600f6a4ef9821309" exitCode=0
Oct 30 22:56:37.816738 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.816624    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" event={"ID":"3e122a52-d9af-417e-92d5-83ac2c2155be","Type":"ContainerDied","Data":"16ed0ab553c68e09dbf6f40e2ce7b88cd435af0476721419600f6a4ef9821309"}
Oct 30 22:56:37.902573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.902545    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:56:37.991262 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.991199    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\" (UniqueName: \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\") pod \"3e122a52-d9af-417e-92d5-83ac2c2155be\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") "
Oct 30 22:56:37.991582 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.991309    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\") pod \"3e122a52-d9af-417e-92d5-83ac2c2155be\" (UID: \"3e122a52-d9af-417e-92d5-83ac2c2155be\") "
Oct 30 22:56:37.991653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.991632    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume" pod "3e122a52-d9af-417e-92d5-83ac2c2155be" (UID: "3e122a52-d9af-417e-92d5-83ac2c2155be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:37.993188 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:37.993169    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz" pod "3e122a52-d9af-417e-92d5-83ac2c2155be" (UID: "3e122a52-d9af-417e-92d5-83ac2c2155be"). InnerVolumeSpecName "kube-api-access-sxbzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:38.092955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.092924    4410 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e122a52-d9af-417e-92d5-83ac2c2155be-plugins-config-volume\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:38.092955 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.092951    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbzz\" (UniqueName: \"kubernetes.io/projected/3e122a52-d9af-417e-92d5-83ac2c2155be-kube-api-access-sxbzz\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:38.822640 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.822604    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"
Oct 30 22:56:38.822803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.822602    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6" event={"ID":"3e122a52-d9af-417e-92d5-83ac2c2155be","Type":"ContainerDied","Data":"de2e4644662891120f1b9b8a72666654b8c6fa76d22685b2064cc42f8196b2c4"}
Oct 30 22:56:38.822803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.822742    4410 scope.go:122] "RemoveContainer" containerID="16ed0ab553c68e09dbf6f40e2ce7b88cd435af0476721419600f6a4ef9821309"
Oct 30 22:56:38.951282 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.951248    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:38.984690 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:38.984662    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:56:39.033928 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:39.033904    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89cf28-66bc8f7c5f-9f6g6"]
Oct 30 22:56:39.374247 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:39.374213    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:41.017974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:41.017881    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" path="/var/lib/kubelet/pods/3e122a52-d9af-417e-92d5-83ac2c2155be/volumes"
Oct 30 22:56:43.558815 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:43.558765    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.122975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:45.122938    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.352691 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:45.352658    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.521792 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:45.521750    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:46.026863 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:46.026811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:50.520112 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:56:50.520074    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:56:50.651626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:50.651587    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:52.056841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.056808    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"]
Oct 30 22:56:52.106540 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.106492    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"]
Oct 30 22:56:52.114340 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.109626    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f066e2cf-x457k"]
Oct 30 22:56:52.116803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.116764    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d90e87-stp6b"]
Oct 30 22:56:52.546853 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.546820    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:52.705889 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.705848    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:52.827122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.827044    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:56:52.827350 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.827323    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy" containerID="cri-o://ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6" gracePeriod=30
Oct 30 22:56:52.879704 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:52.879676    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.017724 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.017689    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293d7c07-547f-4636-9eb4-da2565a6df36" path="/var/lib/kubelet/pods/293d7c07-547f-4636-9eb4-da2565a6df36/volumes"
Oct 30 22:56:53.018137 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.018122    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a91940-6108-4e72-8bfa-3c1ba04f4d5b" path="/var/lib/kubelet/pods/a6a91940-6108-4e72-8bfa-3c1ba04f4d5b/volumes"
Oct 30 22:56:53.281127 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.281090    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.794731 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.794702    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"]
Oct 30 22:56:53.795076 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795063    4410 cpu_manager.go:414] "containerMap: removing container" podUID="d27a1957-8279-456e-b6ca-5244e1fa092b" containerName="revoke-keys"
Oct 30 22:56:53.795126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795077    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27a1957-8279-456e-b6ca-5244e1fa092b" containerName="revoke-keys"
Oct 30 22:56:53.795126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795092    4410 cpu_manager.go:414] "containerMap: removing container" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing"
Oct 30 22:56:53.795126 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795097    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing"
Oct 30 22:56:53.795224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795156    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27a1957-8279-456e-b6ca-5244e1fa092b" containerName="revoke-keys"
Oct 30 22:56:53.795224 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.795166    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e122a52-d9af-417e-92d5-83ac2c2155be" containerName="payload-processing"
Oct 30 22:56:53.799686 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.799666    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:53.855917 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.855892    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"]
Oct 30 22:56:53.902812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.902765    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:53.902812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.902801    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:53.903046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.902857    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:53.903046 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.902950    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:56:53.936244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.936220    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjx4n\" (UniqueName: \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:53.936369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.936262    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:53.936369 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:53.936285    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.037153 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.037130    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjx4n\" (UniqueName: \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.037277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.037172    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.037277 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.037196    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.037857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.037834    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.039445 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.039425    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.093645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.093584    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjx4n\" (UniqueName: \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\") pod \"maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.112402 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.112381    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:54.481456 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.481427    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"]
Oct 30 22:56:54.484009 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:56:54.483982    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181bf3d6_29fb_4260_96ca_d277ab8cd64d.slice/crio-cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919 WatchSource:0}: Error finding container cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919: Status 404 returned error can't find the container with id cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919
Oct 30 22:56:54.910824 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.910794    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2" event={"ID":"181bf3d6-29fb-4260-96ca-d277ab8cd64d","Type":"ContainerStarted","Data":"b783326df141880a6f30c07ca823d3ea7b81f23b05c96ea64963ddcbd18d8efd"}
Oct 30 22:56:54.910979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:54.910843    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2" event={"ID":"181bf3d6-29fb-4260-96ca-d277ab8cd64d","Type":"ContainerStarted","Data":"cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919"}
Oct 30 22:56:55.029521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:55.029433    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2" podStartSLOduration=2.029420325 podStartE2EDuration="2.029420325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:55.02835623 +0000 UTC m=+4468.117094035" watchObservedRunningTime="2026-10-30 22:56:55.029420325 +0000 UTC m=+4468.118158125"
Oct 30 22:56:55.690898 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:55.690868    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:55.917150 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:55.917111    4410 generic.go:356] "Generic (PLEG): container finished" podID="181bf3d6-29fb-4260-96ca-d277ab8cd64d" containerID="b783326df141880a6f30c07ca823d3ea7b81f23b05c96ea64963ddcbd18d8efd" exitCode=0
Oct 30 22:56:55.917318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:55.917193    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2" event={"ID":"181bf3d6-29fb-4260-96ca-d277ab8cd64d","Type":"ContainerDied","Data":"b783326df141880a6f30c07ca823d3ea7b81f23b05c96ea64963ddcbd18d8efd"}
Oct 30 22:56:57.050354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.050329    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:57.163035 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.163007    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\" (UniqueName: \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\") pod \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") "
Oct 30 22:56:57.163209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.163121    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\") pod \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") "
Oct 30 22:56:57.163209 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.163151    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\") pod \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\" (UID: \"181bf3d6-29fb-4260-96ca-d277ab8cd64d\") "
Oct 30 22:56:57.163585 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.163547    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca" pod "181bf3d6-29fb-4260-96ca-d277ab8cd64d" (UID: "181bf3d6-29fb-4260-96ca-d277ab8cd64d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:57.165049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.165026    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n" pod "181bf3d6-29fb-4260-96ca-d277ab8cd64d" (UID: "181bf3d6-29fb-4260-96ca-d277ab8cd64d"). InnerVolumeSpecName "kube-api-access-xjx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:57.165151 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.165135    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca" pod "181bf3d6-29fb-4260-96ca-d277ab8cd64d" (UID: "181bf3d6-29fb-4260-96ca-d277ab8cd64d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:57.264272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.264193    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjx4n\" (UniqueName: \"kubernetes.io/projected/181bf3d6-29fb-4260-96ca-d277ab8cd64d-kube-api-access-xjx4n\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:57.264272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.264219    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:57.264272 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.264231    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bf3d6-29fb-4260-96ca-d277ab8cd64d-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:57.928655 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.928627    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"
Oct 30 22:56:57.928808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.928666    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2" event={"ID":"181bf3d6-29fb-4260-96ca-d277ab8cd64d","Type":"ContainerDied","Data":"cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919"}
Oct 30 22:56:57.928808 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:57.928713    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef685efd9235327a6816bf779156504cce71d2e8d68935559160123ec34b919"
Oct 30 22:56:58.076882 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.076857    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:56:58.170939 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.170916    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.170970    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171001    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171036    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171060    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171105    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171136    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171182    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171218    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\") pod \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\" (UID: \"b4359549-ca53-4e0d-96f0-ad450a2d74c7\") "
Oct 30 22:56:58.171518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171329    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:58.171518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171339    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:58.171518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171298    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:58.171672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171539    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.171672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171558    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.171672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171573    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.171672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171604    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:58.171672 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.171606    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:58.173192 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.173161    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "kube-api-access-nbp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:58.173308 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.173256    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:56:58.173557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.173531    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:58.173557 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.173547    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token" pod "b4359549-ca53-4e0d-96f0-ad450a2d74c7" (UID: "b4359549-ca53-4e0d-96f0-ad450a2d74c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:58.272966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272928    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.272966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272960    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.272966 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272969    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.273211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272979    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.273211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272988    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbp5z\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-kube-api-access-nbp5z\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.273211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.272998    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4359549-ca53-4e0d-96f0-ad450a2d74c7-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.324568 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.324535    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:56:58.337984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.337955    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 22:56:58.345843 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.345814    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 22:56:58.514518 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.514441    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:58.934944 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.934910    4410 generic.go:356] "Generic (PLEG): container finished" podID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerID="ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6" exitCode=0
Oct 30 22:56:58.935103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.934976    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"
Oct 30 22:56:58.935103 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.934973    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerDied","Data":"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"}
Oct 30 22:56:58.935203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.935122    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk" event={"ID":"b4359549-ca53-4e0d-96f0-ad450a2d74c7","Type":"ContainerDied","Data":"d2e35c8c82957b63169492f1a00d085f71eaf7901e28bbf786d6302dfdcfeeab"}
Oct 30 22:56:58.935203 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.935143    4410 scope.go:122] "RemoveContainer" containerID="ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"
Oct 30 22:56:58.944976 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.944958    4410 scope.go:122] "RemoveContainer" containerID="c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"
Oct 30 22:56:58.952980 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.952963    4410 scope.go:122] "RemoveContainer" containerID="ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"
Oct 30 22:56:58.953228 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:56:58.953203    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6\": container with ID starting with ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6 not found: ID does not exist" containerID="ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"
Oct 30 22:56:58.953300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.953239    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6"} err="failed to get container status \"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6\": rpc error: code = NotFound desc = could not find container \"ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6\": container with ID starting with ded2f1d0869bb8cfb73a8898d2047327a9194895047580e8a1ab8a67800ba7b6 not found: ID does not exist"
Oct 30 22:56:58.953300 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.953262    4410 scope.go:122] "RemoveContainer" containerID="c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"
Oct 30 22:56:58.953489 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:56:58.953467    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd\": container with ID starting with c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd not found: ID does not exist" containerID="c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"
Oct 30 22:56:58.953527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:58.953496    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd"} err="failed to get container status \"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd\": rpc error: code = NotFound desc = could not find container \"c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd\": container with ID starting with c631840c5fccb4f73be37fc3428ad0c0e0c9d393d59ba073fa49d78bff55f6dd not found: ID does not exist"
Oct 30 22:56:59.097331 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:59.097299    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:56:59.104086 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:56:59.104056    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e450a-openshift-default-dd9f4d985-xd9dk"]
Oct 30 22:57:00.849281 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:00.849246    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:01.018094 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:01.018054    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" path="/var/lib/kubelet/pods/b4359549-ca53-4e0d-96f0-ad450a2d74c7/volumes"
Oct 30 22:57:02.623768 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:02.623731    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:05.653721 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:05.653690    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:05.816129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:05.816071    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:05.974903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:05.974811    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:09.884099 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884066    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"]
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884395    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884406    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884414    4410 cpu_manager.go:414] "containerMap: removing container" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884420    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884427    4410 cpu_manager.go:414] "containerMap: removing container" podUID="181bf3d6-29fb-4260-96ca-d277ab8cd64d" containerName="revoke-keys"
Oct 30 22:57:09.884485 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884432    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181bf3d6-29fb-4260-96ca-d277ab8cd64d" containerName="revoke-keys"
Oct 30 22:57:09.884695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884514    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884521    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4359549-ca53-4e0d-96f0-ad450a2d74c7" containerName="istio-proxy"
Oct 30 22:57:09.884695 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.884528    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="181bf3d6-29fb-4260-96ca-d277ab8cd64d" containerName="revoke-keys"
Oct 30 22:57:09.887415 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.887394    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:09.978960 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.978929    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:09.979116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.978998    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:09.979116 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.979069    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfpq\" (UniqueName: \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:09.988375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.988345    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:09.988375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.988360    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:57:09.988550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.988377    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:09.988550 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.988409    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:09.990296 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:09.990276    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"]
Oct 30 22:57:10.080122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.080090    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.080293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.080167    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.080293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.080225    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfpq\" (UniqueName: \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.080867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.080838    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.082517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.082492    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.136514 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.136461    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfpq\" (UniqueName: \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\") pod \"maas-api-revoke-sub-dbe71422-cp4bs\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.199919 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.199899    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:10.570338 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:10.570315    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"]
Oct 30 22:57:10.572496 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:57:10.572467    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b77588_dc37_47d0_b637_dc5ccc60a0d1.slice/crio-1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac WatchSource:0}: Error finding container 1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac: Status 404 returned error can't find the container with id 1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac
Oct 30 22:57:11.006836 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:11.006799    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs" event={"ID":"61b77588-dc37-47d0-b637-dc5ccc60a0d1","Type":"ContainerStarted","Data":"447bc74e7728bff7173feffcdb022a389df2ba272504806eb6ce60c7b73b8842"}
Oct 30 22:57:11.007244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:11.006855    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs" event={"ID":"61b77588-dc37-47d0-b637-dc5ccc60a0d1","Type":"ContainerStarted","Data":"1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac"}
Oct 30 22:57:11.120578 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:11.120527    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs" podStartSLOduration=2.120513461 podStartE2EDuration="2.120513461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:11.119148792 +0000 UTC m=+4484.207886595" watchObservedRunningTime="2026-10-30 22:57:11.120513461 +0000 UTC m=+4484.209251313"
Oct 30 22:57:12.014425 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:12.014392    4410 generic.go:356] "Generic (PLEG): container finished" podID="61b77588-dc37-47d0-b637-dc5ccc60a0d1" containerID="447bc74e7728bff7173feffcdb022a389df2ba272504806eb6ce60c7b73b8842" exitCode=0
Oct 30 22:57:12.014827 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:12.014470    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs" event={"ID":"61b77588-dc37-47d0-b637-dc5ccc60a0d1","Type":"ContainerDied","Data":"447bc74e7728bff7173feffcdb022a389df2ba272504806eb6ce60c7b73b8842"}
Oct 30 22:57:13.150108 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.150087    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:13.311072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.311002    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\") pod \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") "
Oct 30 22:57:13.311072 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.311045    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\" (UniqueName: \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\") pod \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") "
Oct 30 22:57:13.311253 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.311084    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\") pod \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\" (UID: \"61b77588-dc37-47d0-b637-dc5ccc60a0d1\") "
Oct 30 22:57:13.311517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.311486    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca" pod "61b77588-dc37-47d0-b637-dc5ccc60a0d1" (UID: "61b77588-dc37-47d0-b637-dc5ccc60a0d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:13.313082 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.313043    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca" pod "61b77588-dc37-47d0-b637-dc5ccc60a0d1" (UID: "61b77588-dc37-47d0-b637-dc5ccc60a0d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:13.313230 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.313211    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq" pod "61b77588-dc37-47d0-b637-dc5ccc60a0d1" (UID: "61b77588-dc37-47d0-b637-dc5ccc60a0d1"). InnerVolumeSpecName "kube-api-access-bkfpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:13.344246 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.344217    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main" containerID="cri-o://7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794" gracePeriod=45
Oct 30 22:57:13.356718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.356578    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main" containerID="cri-o://d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d" gracePeriod=45
Oct 30 22:57:13.363638 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.363611    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main" containerID="cri-o://f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb" gracePeriod=45
Oct 30 22:57:13.411908 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.411887    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-xks-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.411984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.411910    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfpq\" (UniqueName: \"kubernetes.io/projected/61b77588-dc37-47d0-b637-dc5ccc60a0d1-kube-api-access-bkfpq\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.411984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.411920    4410 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b77588-dc37-47d0-b637-dc5ccc60a0d1-maas-api-openshift-ca\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.607813 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.607764    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:57:13.627981 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.627950    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:57:13.637038 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.637019    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 22:57:13.715049 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715023    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\" (UniqueName: \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715064    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715097    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715118    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715175    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715218 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715207    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\") pod \"32cf4881-d14a-4d5d-8455-3d4358009060\" (UID: \"32cf4881-d14a-4d5d-8455-3d4358009060\") "
Oct 30 22:57:13.715522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715487    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.715522 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715488    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.715639 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.715518    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.717146 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.717125    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.717242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.717134    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:13.717242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.717202    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc" pod "32cf4881-d14a-4d5d-8455-3d4358009060" (UID: "32cf4881-d14a-4d5d-8455-3d4358009060"). InnerVolumeSpecName "kube-api-access-cp6mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:13.816434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816353    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.816434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816408    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\" (UniqueName: \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.816434 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816433    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\" (UniqueName: \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816451    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816486    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816509    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816599    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816629    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816651    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816669    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816673    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.816734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816685    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\") pod \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\" (UID: \"fb5444c7-6aa1-41f2-8604-4a6864fd7f55\") "
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816808    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\") pod \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\" (UID: \"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982\") "
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.816816    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817194    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817219    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp6mc\" (UniqueName: \"kubernetes.io/projected/32cf4881-d14a-4d5d-8455-3d4358009060-kube-api-access-cp6mc\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817222    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817236    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817252    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4881-d14a-4d5d-8455-3d4358009060-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817266    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817279    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817289 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817292    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4881-d14a-4d5d-8455-3d4358009060-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817305    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.817867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817672    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.817867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817721    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.817867 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.817843    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.819089 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819055    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "kube-api-access-ws89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:13.819198 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819108    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.819382 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819356    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:13.819457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819436    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "kube-api-access-4vqkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:13.819506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819486    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs" pod "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" (UID: "c62a0376-d5cf-4f17-95b1-f0c6ba5c6982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:13.819957 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.819941    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm" pod "fb5444c7-6aa1-41f2-8604-4a6864fd7f55" (UID: "fb5444c7-6aa1-41f2-8604-4a6864fd7f55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:13.918573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918538    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws89v\" (UniqueName: \"kubernetes.io/projected/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-kube-api-access-ws89v\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918573 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918569    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqkn\" (UniqueName: \"kubernetes.io/projected/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-kube-api-access-4vqkn\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918582    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918595    4410 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-tmp-dir\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918607    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918617    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918627    4410 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-home\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918637    4410 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-dshm\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918649    4410 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5444c7-6aa1-41f2-8604-4a6864fd7f55-model-cache\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:13.918800 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:13.918660    4410 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982-tls-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:14.027182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.027153    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs"
Oct 30 22:57:14.027366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.027139    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe71422-cp4bs" event={"ID":"61b77588-dc37-47d0-b637-dc5ccc60a0d1","Type":"ContainerDied","Data":"1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac"}
Oct 30 22:57:14.027366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.027275    4410 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5d79349dd9a286d3d3abb26ffb2aeeff05f103329a4e96e4e251a053fc4cac"
Oct 30 22:57:14.028746 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.028720    4410 generic.go:356] "Generic (PLEG): container finished" podID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerID="d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d" exitCode=0
Oct 30 22:57:14.028903 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.028873    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" event={"ID":"fb5444c7-6aa1-41f2-8604-4a6864fd7f55","Type":"ContainerDied","Data":"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"}
Oct 30 22:57:14.028974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.028907    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc" event={"ID":"fb5444c7-6aa1-41f2-8604-4a6864fd7f55","Type":"ContainerDied","Data":"2755f725c3fae7ae14019b4051d68fbcc3969a52719bff5d9a5247b4d8c76bc9"}
Oct 30 22:57:14.028974 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.028928    4410 scope.go:122] "RemoveContainer" containerID="d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"
Oct 30 22:57:14.029092 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.029066    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"
Oct 30 22:57:14.031532 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.031511    4410 generic.go:356] "Generic (PLEG): container finished" podID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerID="f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb" exitCode=0
Oct 30 22:57:14.031619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.031578    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" event={"ID":"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982","Type":"ContainerDied","Data":"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"}
Oct 30 22:57:14.031619 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.031601    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z" event={"ID":"c62a0376-d5cf-4f17-95b1-f0c6ba5c6982","Type":"ContainerDied","Data":"50f2a0ae83cdb3e254d1fe9e6b9450c6c0729b7d84e1fbbf974b017557722a0e"}
Oct 30 22:57:14.031750 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.031652    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"
Oct 30 22:57:14.036012 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.035987    4410 generic.go:356] "Generic (PLEG): container finished" podID="32cf4881-d14a-4d5d-8455-3d4358009060" containerID="7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794" exitCode=0
Oct 30 22:57:14.036120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.036096    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"
Oct 30 22:57:14.036120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.036107    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" event={"ID":"32cf4881-d14a-4d5d-8455-3d4358009060","Type":"ContainerDied","Data":"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"}
Oct 30 22:57:14.036242 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.036133    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d" event={"ID":"32cf4881-d14a-4d5d-8455-3d4358009060","Type":"ContainerDied","Data":"7ff6129b077cbd5775dd0a0ed1dc79c5f983c6488e2c22497f88ee59362dc90f"}
Oct 30 22:57:14.041949 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.041916    4410 scope.go:122] "RemoveContainer" containerID="d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"
Oct 30 22:57:14.042235 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:14.042209    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d\": container with ID starting with d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d not found: ID does not exist" containerID="d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"
Oct 30 22:57:14.042319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.042245    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d"} err="failed to get container status \"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d\": rpc error: code = NotFound desc = could not find container \"d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d\": container with ID starting with d81281821cb393cfb9237a956346c95bd56d3cff4e3a5aa5aeab532e95a05f3d not found: ID does not exist"
Oct 30 22:57:14.042319 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.042266    4410 scope.go:122] "RemoveContainer" containerID="f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"
Oct 30 22:57:14.050539 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.050523    4410 scope.go:122] "RemoveContainer" containerID="f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"
Oct 30 22:57:14.050784 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:14.050756    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb\": container with ID starting with f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb not found: ID does not exist" containerID="f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"
Oct 30 22:57:14.050841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.050798    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb"} err="failed to get container status \"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb\": rpc error: code = NotFound desc = could not find container \"f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb\": container with ID starting with f8864d886ad84884d0a5d3115d17dd8923dc9566f3876261efcfc93b6b5f03fb not found: ID does not exist"
Oct 30 22:57:14.050841 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.050814    4410 scope.go:122] "RemoveContainer" containerID="7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"
Oct 30 22:57:14.058348 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.058332    4410 scope.go:122] "RemoveContainer" containerID="7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"
Oct 30 22:57:14.058612 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:14.058594    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794\": container with ID starting with 7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794 not found: ID does not exist" containerID="7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"
Oct 30 22:57:14.058660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.058620    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794"} err="failed to get container status \"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794\": rpc error: code = NotFound desc = could not find container \"7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794\": container with ID starting with 7a79f1000262b86e45844c0db2f96a320b704931b75392506fce46519336b794 not found: ID does not exist"
Oct 30 22:57:14.191182 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.191151    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 22:57:14.194905 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.194882    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/simulator-w3-e90e35-kserve-696f7c57d5-jgl6z"]
Oct 30 22:57:14.355507 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.355476    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:57:14.360988 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.360962    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/distinct-2-w3-e90e35-kserve-d77dd6c44-g9z9d"]
Oct 30 22:57:14.469465 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.469429    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 22:57:14.475660 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:14.475637    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e90e35/premium-w3-e90e35-kserve-748b8d7548-hlnhc"]
Oct 30 22:57:15.017979 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:15.017947    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" path="/var/lib/kubelet/pods/32cf4881-d14a-4d5d-8455-3d4358009060/volumes"
Oct 30 22:57:15.018435 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:15.018412    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" path="/var/lib/kubelet/pods/c62a0376-d5cf-4f17-95b1-f0c6ba5c6982/volumes"
Oct 30 22:57:15.018878 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:15.018860    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" path="/var/lib/kubelet/pods/fb5444c7-6aa1-41f2-8604-4a6864fd7f55/volumes"
Oct 30 22:57:20.520624 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:20.520583    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:57:25.387618 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:25.387580    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:26.876782 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:26.876745    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:27.180378 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:27.180346    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:27.395715 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:27.395679    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"]
Oct 30 22:57:27.444436 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:27.444365    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e90e35-mrct2"]
Oct 30 22:57:27.826206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:27.826118    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:29.018351 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:29.018314    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181bf3d6-29fb-4260-96ca-d277ab8cd64d" path="/var/lib/kubelet/pods/181bf3d6-29fb-4260-96ca-d277ab8cd64d/volumes"
Oct 30 22:57:30.048091 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.048058    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"]
Oct 30 22:57:30.051251 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.051224    4410 scope.go:122] "RemoveContainer" containerID="72b064a77c564f272739d783aabdbd77c735d3c9f6a8a7527c9adb6706d7c979"
Oct 30 22:57:30.059656 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.059638    4410 scope.go:122] "RemoveContainer" containerID="c3aa9ff3e836db9b160524e438417dbfe93fb749663fd9c6d289dee41311cff8"
Oct 30 22:57:30.067265 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.067244    4410 scope.go:122] "RemoveContainer" containerID="6feabbbee2042dd06b7a8dd4b4b33bf0f202da7e11c3fb32e39b0487304d8b82"
Oct 30 22:57:30.074614 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.074591    4410 scope.go:122] "RemoveContainer" containerID="180ec93c56a566df5520c97ee3ab821d573ae45e64d6e6523b3031c420bfe5cf"
Oct 30 22:57:30.081688 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.081672    4410 scope.go:122] "RemoveContainer" containerID="190bbbfc7fffb361a69a2cf1f54634cd42f0fed12bc935635396be0461244b7b"
Oct 30 22:57:30.098258 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.098232    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"]
Oct 30 22:57:30.100599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.100585    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4eaf46-s9lhk"]
Oct 30 22:57:30.102322 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:30.102305    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbbbeae-tgbvj"]
Oct 30 22:57:31.017825 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:31.017792    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0fc698-3975-4f41-9394-18d65c920574" path="/var/lib/kubelet/pods/6d0fc698-3975-4f41-9394-18d65c920574/volumes"
Oct 30 22:57:31.018211 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:31.018197    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b124ed-1e81-4a99-ad73-105af27af79d" path="/var/lib/kubelet/pods/f2b124ed-1e81-4a99-ad73-105af27af79d/volumes"
Oct 30 22:57:33.170079 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170050    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170382    4410 cpu_manager.go:414] "containerMap: removing container" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170394    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170405    4410 cpu_manager.go:414] "containerMap: removing container" podUID="61b77588-dc37-47d0-b637-dc5ccc60a0d1" containerName="revoke-keys"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170410    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b77588-dc37-47d0-b637-dc5ccc60a0d1" containerName="revoke-keys"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170418    4410 cpu_manager.go:414] "containerMap: removing container" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170422    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170437    4410 cpu_manager.go:414] "containerMap: removing container" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170441    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170499    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5444c7-6aa1-41f2-8604-4a6864fd7f55" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170506    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b77588-dc37-47d0-b637-dc5ccc60a0d1" containerName="revoke-keys"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170513    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cf4881-d14a-4d5d-8455-3d4358009060" containerName="main"
Oct 30 22:57:33.170579 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.170520    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62a0376-d5cf-4f17-95b1-f0c6ba5c6982" containerName="main"
Oct 30 22:57:33.175506 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.175484    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.228301 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.228277    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:57:33.277497 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.277443    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\""
Oct 30 22:57:33.279324 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279303    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:33.279529 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279505    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279549    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279607 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279576    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279707    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279751    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279812 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279797    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279918 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279893    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.279984 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.279964    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrqv\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.280061 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.280037    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.380948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.380919    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.380948 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.380949    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.380975    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381004    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381030    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381066    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381156 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381119    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrqv\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381153    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381210    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381366    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381406    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381452 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381435    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381737 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381518    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.381890 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.381869    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.383487 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.383464    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.383630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.383613    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.437496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.437443    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.437727 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.437702    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrqv\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.487901 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.487872    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:33.910286 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.910170    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:33.912390 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:57:33.912366    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf330aee_94d4_472e_b6d2_7f7b6d4e5391.slice/crio-64c22bb4e94afc753737c6b648b4d6f60b1663fb60aa6d5e64433c6408b8467c WatchSource:0}: Error finding container 64c22bb4e94afc753737c6b648b4d6f60b1663fb60aa6d5e64433c6408b8467c: Status 404 returned error can't find the container with id 64c22bb4e94afc753737c6b648b4d6f60b1663fb60aa6d5e64433c6408b8467c
Oct 30 22:57:33.914561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.914531    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:57:33.914626 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.914596    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:57:33.914673 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:33.914625    4410 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:57:34.147309 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.147274    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" event={"ID":"bf330aee-94d4-472e-b6d2-7f7b6d4e5391","Type":"ContainerStarted","Data":"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"}
Oct 30 22:57:34.147521 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.147329    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" event={"ID":"bf330aee-94d4-472e-b6d2-7f7b6d4e5391","Type":"ContainerStarted","Data":"64c22bb4e94afc753737c6b648b4d6f60b1663fb60aa6d5e64433c6408b8467c"}
Oct 30 22:57:34.280292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.280184    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" podStartSLOduration=1.280166921 podStartE2EDuration="1.280166921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:34.276636115 +0000 UTC m=+4507.365373918" watchObservedRunningTime="2026-10-30 22:57:34.280166921 +0000 UTC m=+4507.368904726"
Oct 30 22:57:34.488697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.488658    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:34.493565 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.493537    4410 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:34.560505 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.560440    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:34.780318 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.780274    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:34.799430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.799399    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:34.941443 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:34.941411    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:35.153599 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:35.153558    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:35.155438 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:35.155407    4410 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" secret="" err="secret \"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\" not found"
Oct 30 22:57:35.155576 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:35.155424    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:35.156536 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:35.156517    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:35.309299 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:35.309213    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:36.161533 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:36.161493    4410 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" podUID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" containerName="istio-proxy" containerID="cri-o://38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42" gracePeriod=30
Oct 30 22:57:36.613870 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:36.613752    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:36.847593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:36.847565    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4lgz_746521d2-0f3e-4cd3-bc9f-0389bf2ea65d/kuadrant-console-plugin/0.log"
Oct 30 22:57:39.289493 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:39.289451    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:40.210010 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:40.209975    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:57:41.297916 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.297885    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:57:41.405260 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.405234    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:41.551891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551825    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.551891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551868    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.551891 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551890    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551920    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551959    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.551993    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552055    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552117    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552129    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552156    4410 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\") pod \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\" (UID: \"bf330aee-94d4-472e-b6d2-7f7b6d4e5391\") "
Oct 30 22:57:41.552180 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552177    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:41.552645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552455    4410 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-certs\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.552645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552485    4410 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-workload-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.552645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552529    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:41.552645 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552562    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:41.552876 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.552659    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:41.554432 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.554401    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:41.554674 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.554652    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:57:41.554931 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.554908    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:41.555036 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.554975    4410 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv" pod "bf330aee-94d4-472e-b6d2-7f7b6d4e5391" (UID: "bf330aee-94d4-472e-b6d2-7f7b6d4e5391"). InnerVolumeSpecName "kube-api-access-cfrqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:41.653447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653420    4410 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-podinfo\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653447 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653445    4410 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-credential-socket\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653455    4410 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrqv\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-kube-api-access-cfrqv\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653465    4410 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istiod-ca-cert\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653484    4410 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-envoy\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653493    4410 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-data\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.653649 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:41.653502    4410 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf330aee-94d4-472e-b6d2-7f7b6d4e5391-istio-token\") on node \"ip-10-0-128-19.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:42.191316 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.191281    4410 generic.go:356] "Generic (PLEG): container finished" podID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" containerID="38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42" exitCode=0
Oct 30 22:57:42.191527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.191334    4410 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"
Oct 30 22:57:42.191527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.191366    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" event={"ID":"bf330aee-94d4-472e-b6d2-7f7b6d4e5391","Type":"ContainerDied","Data":"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"}
Oct 30 22:57:42.191527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.191420    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2" event={"ID":"bf330aee-94d4-472e-b6d2-7f7b6d4e5391","Type":"ContainerDied","Data":"64c22bb4e94afc753737c6b648b4d6f60b1663fb60aa6d5e64433c6408b8467c"}
Oct 30 22:57:42.191527 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.191452    4410 scope.go:122] "RemoveContainer" containerID="38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"
Oct 30 22:57:42.201293 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.201272    4410 scope.go:122] "RemoveContainer" containerID="38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"
Oct 30 22:57:42.201585 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:42.201549    4410 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42\": container with ID starting with 38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42 not found: ID does not exist" containerID="38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"
Oct 30 22:57:42.201654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.201598    4410 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42"} err="failed to get container status \"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42\": rpc error: code = NotFound desc = could not find container \"38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42\": container with ID starting with 38babf94b53eff9cc0f76454d3819a999ce38534ca9988bbf312b6b611b28f42 not found: ID does not exist"
Oct 30 22:57:42.342315 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.342281    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:42.405516 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.405479    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:42.454453 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:42.454377    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ct4w2"]
Oct 30 22:57:43.017859 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:57:43.017819    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" path="/var/lib/kubelet/pods/bf330aee-94d4-472e-b6d2-7f7b6d4e5391/volumes"
Oct 30 22:57:50.520864 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:57:50.520821    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:58:13.070641 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:13.070551    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:58:14.290728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:14.290691    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:58:15.623175 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:15.623139    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-069c851f-pwkdr_00c91f42-2504-4749-8cef-8b5e161a9580/revoke-keys/0.log"
Oct 30 22:58:15.839803 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:15.839758    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3f0b9466-mvz2m_8207005e-4af3-47cf-be89-5dd18d231650/revoke-keys/0.log"
Oct 30 22:58:16.047206 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.047174    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-99492467-ffstm_fc1cc120-6383-493a-adc0-16ae80ed2215/revoke-keys/0.log"
Oct 30 22:58:16.058593 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.058559    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"]
Oct 30 22:58:16.107734 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.107699    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"]
Oct 30 22:58:16.109354 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.109324    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99492467-ffstm"]
Oct 30 22:58:16.110479 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.110460    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069c851f-pwkdr"]
Oct 30 22:58:16.472630 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.472599    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dbe71422-cp4bs_61b77588-dc37-47d0-b637-dc5ccc60a0d1/revoke-keys/0.log"
Oct 30 22:58:16.692120 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:16.692097    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dc34ff0e-nzmnv_46a07865-7bb7-41f8-b650-69095e8eca0c/revoke-keys/0.log"
Oct 30 22:58:17.024736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:17.024696    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c91f42-2504-4749-8cef-8b5e161a9580" path="/var/lib/kubelet/pods/00c91f42-2504-4749-8cef-8b5e161a9580/volumes"
Oct 30 22:58:17.025122 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:17.025106    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1cc120-6383-493a-adc0-16ae80ed2215" path="/var/lib/kubelet/pods/fc1cc120-6383-493a-adc0-16ae80ed2215/volumes"
Oct 30 22:58:19.146474 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:19.146446    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4lgz_746521d2-0f3e-4cd3-bc9f-0389bf2ea65d/kuadrant-console-plugin/0.log"
Oct 30 22:58:20.521566 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:58:20.521529    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:58:21.645345 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:21.645314    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-h5t9m_0bc23021-e428-442f-b8e1-093e06020210/kube-auth-proxy/0.log"
Oct 30 22:58:30.134545 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.134518    4410 scope.go:122] "RemoveContainer" containerID="d1d101b3667bb816f1e896e8581846c203dc19b3f30ae9bb78b7aa558bdc1eaf"
Oct 30 22:58:30.142915 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.142867    4410 scope.go:122] "RemoveContainer" containerID="f2a7b85010380a0e47d14753d426e8c2605a71f63523b5800d3b746444996ddf"
Oct 30 22:58:30.150542 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.150520    4410 scope.go:122] "RemoveContainer" containerID="30f7d11ea51cd147486045a176a40a1559d8951d8533cde686a3e7a0f232d4c8"
Oct 30 22:58:30.157525 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.157505    4410 scope.go:122] "RemoveContainer" containerID="e9bf8272c3033e52d2844e45a55982ebc1f4a97e87264b730759fda153b83a77"
Oct 30 22:58:30.164697 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.164676    4410 scope.go:122] "RemoveContainer" containerID="ee4e4d6e50add782c9aedf6ef36f36e93dfeb324d17a6cefe3cdf744cde22e5d"
Oct 30 22:58:30.171509 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:30.171488    4410 scope.go:122] "RemoveContainer" containerID="4ce5eb7249d07f9d5a8bae5c243853926fec914f70cc866310d8cb349ee22ed1"
Oct 30 22:58:39.630295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:39.630257    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:49.599100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:49.599061    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:50.521368 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:58:50.521333    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:58:55.506134 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:55.506101    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:55.710993 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:58:55.710957    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:05.540740 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:05.540706    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b7t59_9b6c706e-df72-4ce3-95d7-17f7b670b864/global-pull-secret-syncer/0.log"
Oct 30 22:59:08.051986 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:08.051950    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"]
Oct 30 22:59:08.101380 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:08.101344    4410 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"]
Oct 30 22:59:08.102753 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:08.102724    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0b9466-mvz2m"]
Oct 30 22:59:08.103833 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:08.103813    4410 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc34ff0e-nzmnv"]
Oct 30 22:59:08.723482 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:08.723450    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log"
Oct 30 22:59:09.017390 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:09.017315    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a07865-7bb7-41f8-b650-69095e8eca0c" path="/var/lib/kubelet/pods/46a07865-7bb7-41f8-b650-69095e8eca0c/volumes"
Oct 30 22:59:09.017761 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:09.017746    4410 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8207005e-4af3-47cf-be89-5dd18d231650" path="/var/lib/kubelet/pods/8207005e-4af3-47cf-be89-5dd18d231650/volumes"
Oct 30 22:59:10.128302 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:10.128276    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-19.us-west-1.compute.internal_fe4301f18ed471dd47244e9704246a79/haproxy/0.log"
Oct 30 22:59:12.863718 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:12.863657    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:13.068244 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:13.068199    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:14.864249 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:14.864200    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:15.132728 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:15.132652    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:17.262998 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.262967    4410 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"]
Oct 30 22:59:17.263396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.263306    4410 cpu_manager.go:414] "containerMap: removing container" podUID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" containerName="istio-proxy"
Oct 30 22:59:17.263396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.263317    4410 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" containerName="istio-proxy"
Oct 30 22:59:17.263396 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.263381    4410 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf330aee-94d4-472e-b6d2-7f7b6d4e5391" containerName="istio-proxy"
Oct 30 22:59:17.266366 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.266346    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.319000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.318971    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"kube-root-ca.crt\""
Oct 30 22:59:17.319000 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.318991    4410 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"openshift-service-ca.crt\""
Oct 30 22:59:17.343342 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.343313    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-sys\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.343461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.343376    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-lib-modules\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.343461 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.343447    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-podres\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.343571 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.343500    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-proc\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.343632 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.343564    4410 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcj7c\" (UniqueName: \"kubernetes.io/projected/14e82065-6cb7-4ffe-9198-9d0cb4e60629-kube-api-access-qcj7c\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.367995 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.367972    4410 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-276rx\"/\"default-dockercfg-pbpfk\""
Oct 30 22:59:17.369517 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.369495    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"]
Oct 30 22:59:17.444679 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444644    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-sys\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.444856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444696    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-lib-modules\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.444856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444759    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-podres\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.444856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444798    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-lib-modules\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.444856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444761    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-sys\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.444856 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444818    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-podres\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.445135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444864    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-proc\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.445135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444885    4410 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcj7c\" (UniqueName: \"kubernetes.io/projected/14e82065-6cb7-4ffe-9198-9d0cb4e60629-kube-api-access-qcj7c\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.445135 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.444917    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e82065-6cb7-4ffe-9198-9d0cb4e60629-proc\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.502736 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.502704    4410 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcj7c\" (UniqueName: \"kubernetes.io/projected/14e82065-6cb7-4ffe-9198-9d0cb4e60629-kube-api-access-qcj7c\") pod \"perf-node-gather-daemonset-4xcfp\" (UID: \"14e82065-6cb7-4ffe-9198-9d0cb4e60629\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.517426 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.517361    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4lgz_746521d2-0f3e-4cd3-bc9f-0389bf2ea65d/kuadrant-console-plugin/0.log"
Oct 30 22:59:17.578496 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.578469    4410 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:17.950954 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:17.950839    4410 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"]
Oct 30 22:59:17.953922 ip-10-0-128-19 kubenswrapper[4410]: W1030 22:59:17.953888    4410 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14e82065_6cb7_4ffe_9198_9d0cb4e60629.slice/crio-7f10610da3df8123dffca3f8673b7a2c3346a26d6bbfd82edcb0dbfb8cd00bfb WatchSource:0}: Error finding container 7f10610da3df8123dffca3f8673b7a2c3346a26d6bbfd82edcb0dbfb8cd00bfb: Status 404 returned error can't find the container with id 7f10610da3df8123dffca3f8673b7a2c3346a26d6bbfd82edcb0dbfb8cd00bfb
Oct 30 22:59:18.648789 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:18.648730    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp" event={"ID":"14e82065-6cb7-4ffe-9198-9d0cb4e60629","Type":"ContainerStarted","Data":"ac4789fc104005e14ca573263cfb6437aed4e54bfc3b8dd793bf78212052f2fe"}
Oct 30 22:59:18.649177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:18.648816    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:18.649177 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:18.648842    4410 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp" event={"ID":"14e82065-6cb7-4ffe-9198-9d0cb4e60629","Type":"ContainerStarted","Data":"7f10610da3df8123dffca3f8673b7a2c3346a26d6bbfd82edcb0dbfb8cd00bfb"}
Oct 30 22:59:18.766515 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:18.766464    4410 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp" podStartSLOduration=1.766452771 podStartE2EDuration="1.766452771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:18.765565973 +0000 UTC m=+4611.854303777" watchObservedRunningTime="2026-10-30 22:59:18.766452771 +0000 UTC m=+4611.855190573"
Oct 30 22:59:19.892125 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:19.892087    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:19.940606 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:19.940566    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:20.521547 ip-10-0-128-19 kubenswrapper[4410]: E1030 22:59:20.521503    4410 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-48tph" containerName="extractor"
Oct 30 22:59:22.047062 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:22.047032    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/alertmanager/0.log"
Oct 30 22:59:22.132295 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:22.132263    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:22.193292 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:22.193266    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:22.465430 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:22.465402    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/config-reloader/0.log"
Oct 30 22:59:22.887805 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:22.887705    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/kube-rbac-proxy-web/0.log"
Oct 30 22:59:23.308074 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:23.308044    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/kube-rbac-proxy/0.log"
Oct 30 22:59:23.726343 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:23.726314    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/kube-rbac-proxy-metric/0.log"
Oct 30 22:59:24.153375 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:24.153346    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/prom-label-proxy/0.log"
Oct 30 22:59:24.576653 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:24.576571    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25a8289d-32a6-4839-a865-643fea03e494/init-config-reloader/0.log"
Oct 30 22:59:24.666332 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:24.666308    4410 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-4xcfp"
Oct 30 22:59:24.670118 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:24.670088    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:26.584703 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:26.584666    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:26.703129 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:26.703091    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-w6rkh_a508794c-35dd-46d6-b7c3-fb2ea20481cc/manager/0.log"
Oct 30 22:59:27.132208 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:27.132167    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-97bpt_818705c7-6050-44b1-b2f7-0bae814576b5/monitoring-plugin/0.log"
Oct 30 22:59:27.558857 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:27.558823    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8bvxb_8bd84e2c-d40a-43a5-9026-55ef6949565b/node-exporter/0.log"
Oct 30 22:59:27.982574 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:27.982533    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8bvxb_8bd84e2c-d40a-43a5-9026-55ef6949565b/kube-rbac-proxy/0.log"
Oct 30 22:59:28.403100 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:28.403076    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8bvxb_8bd84e2c-d40a-43a5-9026-55ef6949565b/init-textfile/0.log"
Oct 30 22:59:28.649591 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:28.649559    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:28.795457 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:28.795375    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-l88st_b0ecfeba-6271-47aa-8025-1cfd2b5f4e53/manager/0.log"
Oct 30 22:59:30.208654 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:30.208626    4410 scope.go:122] "RemoveContainer" containerID="431d94baa61f8ede05e87df22cb9ecbc8fe47ab35fd0b438997e84780fc2968f"
Oct 30 22:59:30.216684 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:30.216658    4410 scope.go:122] "RemoveContainer" containerID="57f864d2deafcaa1d58c166ac9b10895b55de228e6ae2b599af9332e88136c97"
Oct 30 22:59:30.224169 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:30.224148    4410 scope.go:122] "RemoveContainer" containerID="3bd7983757be351a59df37b63fab1ecc1976276aeaf23d6095be71a402c15f4a"
Oct 30 22:59:30.231635 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:30.231617    4410 scope.go:122] "RemoveContainer" containerID="bec1c105eec009cfe8efa01433d6eb65b1b2f0235ebcd72079c3d3fac064182f"
Oct 30 22:59:32.710270 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:32.710236    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/prometheus/0.log"
Oct 30 22:59:33.189594 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:33.189562    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/config-reloader/0.log"
Oct 30 22:59:33.613053 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:33.612958    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/thanos-sidecar/0.log"
Oct 30 22:59:34.035131 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:34.035100    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/kube-rbac-proxy-web/0.log"
Oct 30 22:59:34.456850 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:34.456812    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/kube-rbac-proxy/0.log"
Oct 30 22:59:34.876992 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:34.876856    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/kube-rbac-proxy-thanos/0.log"
Oct 30 22:59:35.198676 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:35.198634    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-h5t9m_0bc23021-e428-442f-b8e1-093e06020210/kube-auth-proxy/0.log"
Oct 30 22:59:35.297562 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:35.297533    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cab40cc-a828-481e-b0a6-a5260b0a1cde/init-config-reloader/0.log"
Oct 30 22:59:35.446104 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:35.446074    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-h5t9m_0bc23021-e428-442f-b8e1-093e06020210/kube-auth-proxy/0.log"
Oct 30 22:59:36.571128 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:36.571089    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qprwm_1da1b6c5-5490-4b39-a6c5-c605cb9e6cc2/prometheus-operator-admission-webhook/0.log"
Oct 30 22:59:36.998561 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:36.998527    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755cbfd578-gd5zr_1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e/telemeter-client/0.log"
Oct 30 22:59:37.416756 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:37.416728    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755cbfd578-gd5zr_1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e/reload/0.log"
Oct 30 22:59:37.835186 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:37.835113    4410 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755cbfd578-gd5zr_1423c6ea-f9b5-481c-85e4-1e3eb1f7d74e/kube-rbac-proxy/0.log"
Oct 30 22:59:39.973975 ip-10-0-128-19 kubenswrapper[4410]: I1030 22:59:39.973881    4410 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9l2mj_7524bf5c-af7d-4d94-a660-06fe000da0ae/konnectivity-agent/0.log" limit=1024